Your AI strategy will change. Your evaluation system should not

An AI prototype can impress a room and still tell a leadership team very little. The demonstration proves that the system can produce a useful answer once. It does not prove that it will do so consistently under messy inputs, after a prompt change, when a source is missing, when a tool fails or when the model behind it is replaced.

That distinction is moving from technical hygiene to strategic importance. In a 2024 Google-Datadog discussion, Datadog VP of Engineering Sajid Mehmood made the durable point captured below. The source is commercially interested, so it is better treated as a prompt than as proof. But the observation has aged well.¹

"The piece that's not going anywhere is the evaluation side."

Sajid Mehmood, VP of Engineering, Datadog

By 2026, the US National Institute of Standards and Technology was warning that pre-deployment tests are not enough. Live systems need post-deployment monitoring because real-world inputs, model non-determinism and unexpected consequences can reveal behaviours that controlled testing misses.²

The central argument is simple: evaluation should not be treated as final quality assurance. It should be designed as the operating spine of enterprise AI - the system through which objectives are defined, evidence is retained, changes are compared and failures become institutional learning.

Your most visible AI choices have the shortest shelf life

Enterprise AI discussions are dominated by model names, vendors and methods. Those choices matter, but they are also the most mobile part of the system. Prompting gives way to retrieval. Retrieval is reshaped by larger context windows. Single assistants become tool-using workflows. Workflows become agents. Model families improve, merge or disappear.

The 2024 source itself questioned whether retrieval-augmented generation would remain as important as context windows expanded. The correct response is not to predict which technique wins. It is to avoid making the organisation's standard of evidence dependent on any one technique.

When confidence is tied to a particular stack, every upgrade reopens the argument. A team that knows only that "the new model seems better" has no reliable basis for switching. It must repeat subjective testing, renegotiate trust and hope that gains in one area have not caused regressions elsewhere.

An evaluation system changes the economics of change. The same use cases, representative cases, failure categories and outcome measures can be run against the old and new configurations. The suite should evolve, but it provides continuity. The model becomes a replaceable component; the organisation's definition of acceptable performance remains visible.

A dashboard is not an evaluation system

The language around AI quality is often blurred. Three layers are related, but they do different jobs.

A useful distinction

Technical observability can show that a system was available, fast and inexpensive. It cannot, by itself, show that a purchasing recommendation was correct, that a customer was treated fairly or that a summary preserved the one qualification on which a decision depended. A dashboard may be healthy while the work is poor.

OpenTelemetry is now standardising generative-AI telemetry around model calls, token use, retrieval, tool calls, workflows and evaluation scores across providers. That is a useful portability signal: the evidence needed to inspect AI systems need not live entirely inside one vendor's dashboard. But instrumentation only creates evidence. It does not decide which outcomes matter, which failures are tolerable or who should act when a threshold is crossed.³

Benchmarks need similar caution. NIST's 2026 work distinguishes performance on a fixed benchmark from performance across the broader population of similar tasks. A leaderboard score can be informative without being a warranty for a particular workflow, user population or risk level.⁴

Complexity raises the evidence burden

A bounded summarisation tool has a relatively simple shape: input, transformation, output and human comparison with the source. An agentic system can operate over many turns, select tools, retrieve data, alter external state, adapt to intermediate results and decide when it is finished. The useful capabilities are also new places for mistakes to propagate.

Anthropic's 2026 guidance on agent evaluation separates the trace - the record of messages, tool calls and intermediate steps - from the outcome, the state that actually exists when the work ends. An agent may report that a task is complete while the external record shows that it is not. Effective evaluation therefore combines deterministic checks, model-based judgement and human review rather than trusting the final sentence alone.⁵

Google's Director of AI, Dr Ali Arsanjani, advised organisations to aim for a maturity level "commensurate with their existing capabilities". That is not an argument against agents. It is an argument that permission should increase with inspectability.¹

A system allowed to draft an internal note can tolerate a different failure pattern from one allowed to send the note, approve a refund, change a supplier record or place an order. Each step towards autonomy adds questions: Did it choose the right tool? Use only permitted data? Recover from a failed call? Stop when uncertain? Escalate the right case? Leave an auditable final state?

Arsanjani's companion requirement - "controls and human evaluation at each stage of the game" - becomes more important as systems gain the ability to act rather than merely answer.¹

Average performance hides the failures leadership cares about

The managerial task is not to demand perfection. It is to define where good enough is genuinely good enough - and where it is not. A creative first draft may be useful at 80 per cent quality because a skilled person will reshape it. A regulatory filing may require near-zero tolerance for a particular class of omission. The acceptable threshold belongs to the use case, not to the model in the abstract.

That means separating ordinary quality, high-severity failure, review rate, cost, latency and business outcome. A system can improve its average score while becoming worse on the rare cases that matter most. Severity classes and stop conditions are therefore as important as the headline pass rate.

Build the evaluation spine before the agent

The AI Evaluation Spine

Before an AI system is allowed to scale, five linked questions should have recorded answers. The artefact is deliberately model- and vendor-neutral.

This is not another maturity model or a procurement checklist. It is a handrail around change. Each element should have an owner and a record. Together they allow a steering group to ask whether a proposed model, prompt, data source or agent architecture improves the work without reopening first principles.

The spine also reveals when a project is not ready to scale. If nobody can state the business outcome, assemble representative cases, distinguish severe from tolerable errors or reconstruct a consequential result, the missing capability is not a more powerful model. It is a clearer operating system.

Start where evaluation is cheap

One of the strongest practical observations in the Google-Datadog source concerns summarisation. It is often valuable precisely because the original material remains available for comparison. Incident summaries, meeting handovers and case-note condensation give users a visible basis for trust and give teams a relatively inexpensive way to learn how evaluation works.¹

This is a better starting point than choosing the most impressive demonstration. Select a bounded workflow in which the evidence can be inspected, the consequences are understood and human correction remains easy. Then make the evaluation cycle explicit.

A practical first cycle

1. Choose a bounded workflow. Prefer a task whose source material and expected result can be inspected.
2. Collect real examples. Include ordinary cases, awkward cases and rare failures that would be costly if mishandled.
3. Define good and unacceptable. Turn expert judgement into examples, rubrics and explicit severity classes.
4. Run repeated trials. Probabilistic output varies; one successful demonstration is not a performance measure.
5. Deploy with limits and live sampling. Set permissions, review triggers and a routine for examining real outputs and outcomes.
6. Feed exceptions back. Every new failure should improve the test set, the operating rule or both.

OpenAI's 2025 business guide recommends reviewing roughly 50 to 100 early outputs to expose a useful error taxonomy, then continuing to measure real outputs and incorporate user signals after launch. The number is not sacred; the operating principle is. Early review should produce a living body of cases and judgements, not a one-off approval meeting.⁶

The evaluation system is an organisational asset

Technical teams can instrument a system, but they cannot define commercial acceptability in isolation. The people closest to customers, operations, risk and domain practice know which omissions are fatal, which variations are harmless and what a genuinely useful result looks like. Evaluation turns some of that tacit judgement into a shared asset.

Over time, the representative cases, rubrics, exception history and expert decisions become difficult to copy. Competitors can buy access to the same model. They cannot instantly reproduce the organisation-specific record of what good looks like, how failure presents itself and which trade-offs the business has chosen.

That asset should be portable. Retain test cases, expected outcomes, severity definitions, change logs and ownership outside a single vendor's product. Industry telemetry standards can help, but the more important portability is managerial: the ability to explain why a system was accepted, what evidence supported the decision and how the standard will be applied after the next change.

Durable does not mean static. NIST notes that post-deployment monitoring methods and terminology are still developing, while model providers emphasise that evaluation suites themselves need maintenance. Business objectives change; users find new edge cases; a previously useful test can become stale. The lasting capability is not a frozen benchmark. It is the discipline of defining, measuring, learning and revising.²⁵⁶

Build the capability that survives the upgrade

No leadership team can choose the final AI architecture. Models will improve, retrieval patterns will be reshaped, agent systems will gain new permissions and today's preferred vendors will change position. The better strategy is to build the capability that makes each change testable.

Before approving the next agent, ask five questions: What outcome must improve? What evidence would demonstrate success? Can a consequential result be reconstructed? Which errors are tolerable? Can the same standard be rerun after the system changes?

Your AI strategy will change. If the evaluation system is doing its job, that will be a manageable fact - not a strategic crisis.