The AI audit trail needs a 'why' layer

Executive view

• AI observability is becoming an operating requirement. Organisations increasingly need to know which model ran, what information it retrieved, which tools it called, how long the task took, what it cost and where the system failed.
• That technical trace is necessary. It is not the whole accountability record.
• When an AI output informs a consequential recommendation or action, a later reviewer may also need to know why the result was accepted, which criteria and assumptions mattered, which alternatives were rejected, where a human intervened and who owned the final judgement. This article calls that missing layer decision provenance.
• The practical objective is not to preserve an AI model's private chain of thought. It is to create a risk-proportionate, external decision record that a competent person can inspect, challenge and revisit.

Illustrative organisational vignette

Six months after a supplier was selected, the relationship is failing.

The procurement team can reconstruct the AI-assisted analysis in impressive detail. The logs show which model was used, which documents were retrieved, which scoring tool was called, what recommendation appeared and when a manager approved it.

But the board is asking different questions.

Why was Supplier A preferred to Supplier B? Which assumptions about delivery capacity were decisive? Was the lower price allowed to outweigh a weak service record? Which risks did the AI identify? Which did the manager discount? What new evidence would have changed the decision?

The organisation has a technical audit trail. It does not yet have an adequate account of the judgement.

That distinction will matter more as AI moves from helping people produce work to helping organisations choose, prioritise, recommend and act.

Observability is becoming an operating requirement

In a Google Cloud guide produced with Datadog, Google's Director of AI, Dr Ali Arsanjani, identifies "the emphasis on observability at every stage of the life cycle" as an important trend. Datadog's Sajid Mehmood argues that, even as specific methods change, evaluation is the part that is "not going anywhere".[1]

The practical logic is strong. An AI application can contain prompts, retrieval systems, model calls, tools, filters, databases and human interventions. A failure may originate in the model, the data, the prompt, a tool call, a policy, a handover or the surrounding application. Without traces, logs, evaluations and operational metrics, teams are left guessing.

Datadog's example is especially useful. Its Bits AI system can present relevant telemetry, logs and traces alongside a conclusion so that a user can see the evidence behind it.[1] This turns a bare answer into something more inspectable.

For engineering teams, that can reveal what the system retrieved, which route it followed and where the application broke. It also supports the familiar operating questions: Is the output accurate enough? Is latency acceptable? Is cost rising? Are failures clustering around a particular model, tool or workflow?

As AI systems become more autonomous and more deeply embedded in operations, this visibility stops being a specialist convenience. It becomes part of organisational control.

But it still does not necessarily answer the question in the title.

The black box does not disappear when the logs are complete

Dario Amodei, Anthropic's chief executive, has described the deeper interpretability problem bluntly: with generative AI, "we have no idea, at a specific or precise level, why it makes the choices it does".[2]

A full application trace does not solve that scientific problem. It may show the model's inputs, outputs and tool calls without revealing the precise internal cause of a particular choice.

Nor does a fluent explanation generated afterwards settle the matter. A model can produce a plausible account of its answer without that account becoming verified evidence of its internal mechanism. The safest organisational record is therefore not an AI autobiography.

It is a defensible external rationale.

Microsoft Chief Scientific Officer Eric Horvitz captured the human expectation in a 2020 discussion about AI-supported recommendations: "People want explanations. They don't want to look at a black box with an indicator on it."[3]

That expectation becomes sharper when somebody is expected to accept responsibility for what happens next. The question is no longer only, "Why did the model produce this output?" It is also, "Why did the organisation rely on it?"

Those are related questions, but they are not the same.

A technical trace is not a decision record

Consider the supplier vignette again.

A technical trace might establish:

• the input documents and data
• the model and application version
• the retrieval and tool-call sequence
• the generated score and recommendation
• the timestamp, cost, latency and system errors
• the identity of the person who clicked approve.

That is valuable. It can help reproduce the event, diagnose defects and test whether the same system behaves consistently.

A decision record needs additional information:

• the purpose of the decision and the authority being exercised
• the criteria and constraints applied
• the assumptions that materially shaped the recommendation
• the alternatives considered and why they were rejected
• the evidence the human decision-maker found decisive
• any disagreement, override or qualification
• the uncertainty that remained
• the conditions that should trigger review or reversal.

A log can prove that approval happened. It may not explain why approval was justified.

Call this decision provenance: enough preserved context to reconstruct why a consequential result was accepted as a basis for action.

This is not an argument for documenting every low-risk AI interaction. A routine summary or formatting task may need little beyond ordinary quality control. The depth of the record should rise with the consequence, autonomy, contestability and difficulty of reversing the decision.

But where an organisation may later need to defend, challenge, transfer or revisit a judgement, recording only the machine activity creates a predictable gap.

Governance guidance already recognises the 'why'

The need for rationale is not absent from responsible-AI thinking. The more interesting danger is that it can disappear during implementation.

The UK's Information Commissioner's Office advises organisations to document how system specifications were determined, which alternatives were considered and why they were not chosen. It also asks who is responsible for explanations, what trade-offs were made and why particular explanation types were prioritised.[4]

NIST's AI Risk Management Framework Playbook combines technical instrumentation with organisational decisions. It recommends histories and audit logs, but also the documentation of human oversight, overrides, complaints, escalations and go/no-go decisions made by accountable parties. Elsewhere, it asks organisations to record criteria, assumptions, trade-offs and system limitations.[5]

The UK's Algorithmic Transparency Recording Standard is similarly explicit. It is designed to show how and why public-sector organisations use algorithmic tools, and to identify the senior role ultimately accountable for the tool in operation.[6]

For systems within the EU AI Act's high-risk scope once the relevant phased provisions apply, Article 12 requires automatic event logging. Article 14 separately requires human overseers to understand limitations, interpret outputs and, where appropriate, disregard, override or reverse them.[7]

These are not identical requirements. Logging helps make the system traceable. Human oversight requires a person with sufficient information, authority and competence to exercise judgement.

The principles therefore point in the right direction. The implementation risk is more mundane: technical teams build what can be captured automatically, while the rationale for human acceptance remains in a meeting, an email, a passing comment or nowhere at all.

Four layers of an accountable AI record

A useful operating model separates four layers.

An editorial operating model. Technical traceability, decision provenance and accountability answer different questions.

The layers support different tasks.

Logging and traceability help engineers reproduce and diagnose system behaviour. Decision provenance helps managers, auditors, regulators, colleagues and affected people understand the basis on which the organisation acted. Accountability identifies the human or institutional authority that owns the judgement and the consequences.

A mature system may generate much of the first two layers automatically. The third and fourth often need to be designed into the workflow deliberately.

That design does not always require a long narrative. Structured fields can do much of the work: decision purpose, criteria applied, material assumptions, alternatives, human changes, residual uncertainty, owner and review trigger. The important point is that the information is captured at the decision point, while the evidence and disagreement are still visible.

Do not manufacture the rationale afterwards

A weak 'why' layer can be worse than none if it creates false confidence.

After a decision has failed, it is easy to produce a tidy explanation that makes the outcome look more deliberate than it was. An AI system can make that retrospective story especially polished. The result may resemble accountability while merely laundering hindsight into a plausible narrative.

The record should therefore distinguish contemporaneous evidence from later interpretation.

At the time of the decision, capture the criteria, material assumptions, human changes, unresolved uncertainty and reason for acceptance. If the explanation is later amended, preserve the change rather than silently replacing the original account.

This also avoids an unhelpful demand for raw chain-of-thought. Organisations do not need a verbatim transcript of hidden model reasoning in order to create a useful accountability record. They need the externally defensible basis for action: what evidence was relied upon, what rules or criteria were applied, what alternatives mattered, where human judgement entered and who accepted responsibility.

The objective is not omniscience. It is inspectability.

Scale turns record design into an operating-model question

IBM Chairman, President and Chief Executive Officer Arvind Krishna said in May 2026 that organisations need to "manage AI-driven systems with the same rigor, governance, and scale as their most critical infrastructure".[8]

IBM is selling the infrastructure that supports that proposition, so the statement should be treated as a market signal rather than neutral proof. The direction is nevertheless important. Once an organisation has many models, agents, tools and teams acting across interdependent workflows, accountability cannot depend on one conscientious manager remembering to write a note.

The record has to become part of the operating model.

That usually means combining machine-generated technical evidence with a structured decision checkpoint. High-consequence actions may require explicit human acceptance and rationale. Repeated low-risk decisions may use predefined criteria, sampled review and exception records. Overrides should be visible. Material disagreements should survive. Review triggers should be attached to the decision rather than rediscovered after failure.

The best design is unlikely to be one universal dossier. It will be a tiered system in which the depth of the record matches the risk and reversibility of the action.

The AI Decision Record Test

Use these seven questions before treating an AI audit trail as an accountability record.

An Andrew's Intel decision aid for consequential AI-supported action; risk-proportionate prompts, not an external standard or validated score.

Diagnostic line: If the record answers only questions 1 and 2, the organisation probably has a technical trace, not a complete accountability record.

The test can also be used in vendor and internal-team conversations. Ask not only whether the platform records prompts, responses and tool calls, but whether the surrounding workflow can capture the criteria, human judgement, approval basis and review trigger that make the eventual decision defensible.

Accountability needs a record of judgement

The growth of AI observability is good news. Organisations need the telemetry, logs, traces and evaluations that reveal how complex AI applications behave. Without them, failures remain hidden and improvement becomes guesswork.

But an accountable organisation needs to preserve more than machine activity.

It needs to know what happened, how it happened, why the result was accepted and who owned the judgement. The first two can often be instrumented. The latter two must be designed.

Fei-Fei Li has expressed the enduring principle well: "However autonomous our technology becomes, its impact on the world, for better or worse, will always be our responsibility."[9]

Responsibility without a recoverable rationale is fragile. It leaves the accountable person holding a conclusion without the judgement that made it defensible.

The AI audit trail therefore needs a 'why' layer. Not a theatrical explanation and not an imagined transcript of the model's mind, but a clear record of the evidence, criteria, alternatives, human intervention and authority behind consequential action.

That is what turns observability into accountability.