Evals are the missing
management layer

A polished demonstration can make an AI system look ready. The examples are tidy, the answers are fluent and the failure cases are nowhere in sight. Yet a demonstration usually answers only one question: can the system produce an impressive result under favourable conditions?

Management needs different answers. What does acceptable performance mean in this workflow? Which failures matter most? What evidence is enough to release, expand or stop the system? Who decides? How will the organisation notice that behaviour has changed?

Evals—structured evaluations of model or system behaviour—provide the missing layer between an AI capability and the management decisions around it. They translate intent, risk tolerance and operational judgement into repeatable evidence.

A convincing prototype is not a managed system

AI performance is variable. Inputs change, models change, retrieval sources move, prompts are revised and users find paths that the original designers did not anticipate. Average performance can also conceal a small class of failures with disproportionate consequences.

That makes occasional review and generic benchmark scores too weak for operational management. A benchmark can say something useful about a model under a defined test. It cannot tell an organisation whether its own workflow is safe, reliable or valuable enough to use.

Management begins when the organisation states what the system must do, what it must not do and what evidence will alter a decision.

What an eval actually does

A practical eval connects four elements:

A representative case. An input, trace or scenario drawn from the work the system will actually encounter.

A criterion. A specific property that matters: factual accuracy, policy compliance, appropriate tool use, complete citation, safe escalation or another observable requirement.

An evaluator. A human reviewer, deterministic check, model-based judge or combination that applies the criterion.

A gate or interpretation. The decision attached to the result: release, investigate, escalate, redesign, limit use or stop.

The last element is what turns evaluation into management. A score without a decision rule may improve engineering work, but it does not tell the organisation what to do.

Why evals belong to management

They force the organisation to specify the work

Designing an eval exposes vague ambitions. “Improve customer service” is not testable. “Draft a response that answers the customer’s stated question, uses only approved policy sources, flags uncertainty and routes regulated cases to a named reviewer” is much closer.

They make risk choices visible

Teams often agree that quality matters while disagreeing about which failures are tolerable. An eval set makes those choices explicit. A harmless stylistic flaw should not be treated like an invented compliance statement. A correct final answer reached through an unauthorised tool may still be an unacceptable result.

They assign decision rights

Someone must approve the criteria, accept residual risk, interpret disputed results and decide when the system can progress. Evals make those roles harder to leave implicit.

They create institutional memory

Every production failure, near miss and edge case can become a new test. The eval suite then records what the organisation has learned—not as a retrospective report, but as a living part of the operating system.

A good eval set is a compact record of what the organisation cares about and what experience has taught it not to trust.

Start with failure, not framework

The fastest way to create an impressive but useless eval programme is to begin with a platform, leaderboard or broad metric. Begin instead with failure.

Ask what a bad result looks like in this workflow. Which errors would mislead a customer, create rework, breach a rule, expose confidential information, waste money or weaken a professional decision? Which failures are merely inconvenient, and which should block release?

This produces an error taxonomy grounded in consequences. It also prevents an average score from allowing frequent success to wash out rare but material failure.

Benchmarks are not your workflow

General benchmarks are useful for comparing capabilities and tracking technical progress. The management question is narrower: does this system behave acceptably in our work, with our documents, policies, tools, users and failure costs?

That requires representative cases. A useful evidence set may combine ordinary examples, difficult edge cases, known historical failures, adversarial probes and cases where the correct action is to refuse or escalate.

The set should not be frozen. New products, policies, suppliers and usage patterns change the work. A useful eval suite evolves with them.

The score is not the outcome

Evals verify behaviour under defined conditions. They do not by themselves validate business value. A system can pass a quality threshold and still fail to save time, improve service, reduce risk or earn adoption.

Spotify’s engineering team describes a useful distinction: evals can screen and verify changes before or alongside experiments, while experiments test the effect on users and the business. [4] Operational measures, user research and outcome data remain necessary.

The management stack therefore needs both:

Behaviour evidence: Does the system meet the defined quality, safety and process requirements?

Outcome evidence: Does using it improve the result that justified the investment?

Confusing the two creates a familiar failure: a system that scores well but does not improve the work.

Agents make the evaluation problem wider

An agent can choose tools, retrieve information, plan a sequence, take intermediate actions and decide when to stop. Evaluating only the final answer misses much of the behaviour that creates risk.

The evaluation may need to examine the trajectory: which source was retrieved, whether the right tool was selected, whether permissions were respected, whether an irreversible action was attempted and whether the system escalated when evidence was insufficient. Google’s Agent Development Kit and Anthropic’s guidance both emphasise this wider view of agent behaviour. [3, 7]

A plausible answer reached through the wrong path is not always a successful run.

Automating the judge creates a second control problem

Model-based evaluators can review far more examples than a human team. They can classify, score against a rubric and provide explanations. That makes them valuable—but not independent truth machines.

An LLM-as-judge can inherit bias, favour a position or style, misunderstand the rubric and drift when the evaluator model changes. Research into position bias across 15 LLM judges and more than 150,000 instances found that the effect is not random; it varies by judge and task. [8]

Automated judges therefore need their own controls:

a human-labelled reference set;

calibration against that set before operational use;

regular sampling and audit of decisions;

clear treatment of disagreement, uncertainty and borderline cases;

version control for prompts, rubrics and evaluator models.

The point is not to reject automated evaluation. It is to manage it as another model-assisted decision process.

An illustrative board-paper vignette

Imagine an AI assistant that drafts short board-paper summaries. In a demonstration it produces concise, confident prose. The organisation could evaluate it with a generic “helpfulness” score. That would miss the failures that matter.

A management-led eval might instead test whether the draft identifies the decision requested, distinguishes fact from management judgement, preserves material caveats, cites the approved paper, avoids introducing unsupported claims and escalates when annexes conflict.

The release gate might require zero unsupported numerical claims in a human-reviewed sample, high recall for decisions and risks, and mandatory escalation on contradictory source material. A separate experiment would then test whether the assistant reduces preparation time without increasing review effort or weakening board understanding.

The eval does not merely score the prose. It expresses how the organisation expects the work to be done.

The Management Eval Canvas

Six decisions are enough to move from an interesting prototype toward a managed evaluation discipline.

On a narrow screen, each row is shown as a separate card.

Start small enough to learn

An organisation does not need a vast evaluation platform before it can begin. A disciplined first cycle can be compact:

Choose one bounded workflow with a named owner.

Review roughly 50–100 real or representative outputs and traces.

Create an error taxonomy from what actually goes wrong.

Build three to five high-value evaluators around the most consequential failures.

Create a human-labelled reference set and calibrate any automated judge.

Define release gates, escalation rules and the person who can accept the result.

Sample live operation and add new failures to the suite.

Airbnb’s account of eval-driven development describes a similar practical starting point: small representative datasets, a few focused evaluators, calibration on dozens of examples and human judgement as the final decision-maker. [1]

Seven questions leaders should ask

1. Which decision changes if this eval result changes? If no decision changes, the measure may be decorative.

2. Which failure class matters most? Average quality should not hide the error that carries the real consequence.

3. How representative is the evidence set? Know whose work, language, edge cases and conditions are missing.

4. Are we evaluating the path as well as the answer? This is essential when tools, retrieval or autonomous actions are involved.

5. Who calibrated the evaluator? A model-based judge needs evidence that it agrees with informed human judgement often enough for the intended use.

6. What happens when results disagree? Borderline cases need an escalation route, not forced certainty.

7. How does production experience change the suite? An eval programme that does not learn from live failures will become stale.

The missing layer is not another dashboard

A dashboard can display scores. The management layer decides why those scores matter, what evidence sits behind them, which trade-offs are acceptable and who has authority to act.

NIST’s AI Risk Management Framework places measurement inside a wider cycle of govern, map, measure and manage. [5, 6] That is the right orientation. Evaluation is not a one-off test at the end of development. It is part of how an organisation governs a changing system throughout its life.

Evals do not remove judgement. They decide where judgement lives, who owns it and how the organisation learns.