The AI can be accurate and still make the decision worse

Imagine a manager reviewing a programme that helps customers avoid falling into arrears. The AI has become good at predicting which accounts will pay late. The obvious next move appears to be directing all the team’s attention towards the highest-risk accounts.

But suppose some of those customers cannot be helped by the intervention available, while a timely conversation would prevent difficulties for others further down the ranking. The prediction can be correct and the allocation poor. Even a perfect list of who will struggle would not, by itself, reveal whom the team can help.

This is a hypothetical example, not a claim about a particular lender. It exposes a distinction that matters whenever a forecast starts directing action: prediction accuracy is not decision value.

Arvind Narayanan and Sayash Kapoor make that distinction explicit in AI Snake Oil. Their chapter on predictive AI includes a section titled “A Good Prediction Is Not a Good Decision”. The title is worth remembering before the next demonstration of a model that promises to improve decisions by making more accurate predictions. ¹

A prediction may contain the effect of the help already being given

The book uses a pneumonia example whose underlying research reveals the trap particularly clearly.

In a 2015 paper revisiting earlier pneumonia-risk research, Rich Caruana and colleagues describe models learning that patients with asthma had a lower observed risk of dying. This reflected the data: those patients tended to receive unusually intensive treatment, which improved their outcomes. It did not mean asthma made an untreated patient safer. ²

Using that pattern to justify withholding admission would remove the very care helping to produce the favourable result. The researchers did not report deploying the suspect neural-network model and then observing avoidable deaths. They judged it too risky to use in the planned clinical trial. The example is a warning caught during evaluation, not evidence of a deployment disaster. ²

That distinction strengthens the management lesson. A model can reproduce a pattern under the old operating arrangements without telling you what will happen after you change those arrangements.

There is an everyday organisational equivalent. A supplier appears dependable because a procurement team repeatedly intervenes to prevent delays. A project stays on schedule because a senior manager quietly resolves its conflicts. A service rarely suffers an outage because maintenance is unusually intensive.

Those are illustrative possibilities, not findings about a particular organisation. In each, apparent low risk may be partly the result of effective attention. A model that recommends removing that attention can mistake successful management for evidence that management was unnecessary.

The question to ask is therefore not only “How accurate was the forecast?” It is also “What actions helped produce the outcomes on which it was trained?”

The target can be measured well and still be wrong for the job

A different problem arises when the model predicts an available measure rather than the outcome the organisation actually cares about.

Ziad Obermeyer and colleagues examined a healthcare algorithm used to identify people for additional help. Their 2019 study found that the system predicted healthcare spending, which was being used as a stand-in for need. At a given risk score, Black patients were substantially sicker than White patients. Unequal access and spending meant that accurately predicting cost did not identify equal levels of illness. ³

The problem was not solved by making the cost prediction more accurate. The researchers showed that the choice of prediction target was central to the disparity and that changing the target could reduce it. ³

This was a study of a particular system, not a verdict on every healthcare algorithm. Its wider lesson is the distinction between the measure and the mission.

In other settings, the same question would be whether the model is predicting an easily recorded event that only imperfectly represents the desired result. Is a closed service ticket a resolved problem? Is staying in a job evidence of doing it well? Is a low number of complaints evidence of good treatment, or merely of a difficult complaints process?

These are not interchangeable outcomes. Before asking a technical team to improve the score, management has to decide what the score is meant to stand for and where that substitution breaks down.

Better predictions do not choose the intervention

Return to the hypothetical arrears team. Its intended outcome might be fewer customers entering serious difficulty, subject to fair treatment and a fixed staffing budget.

A risk score can help describe the problem. It does not automatically answer three further questions: what assistance is available, which customers are likely to benefit from it, and what happens to the people who receive less attention as a result.

The relevant comparison is between possible courses of action. A team could make early calls, offer different repayment arrangements, correct billing problems or refer customers to another service. Each intervention has different costs, constraints and potential effects.

A useful model might support those choices. But a model designed only to predict late payment has not thereby supplied evidence about the effects of those interventions.

Nor should this become a simplistic instruction to help only the easiest cases. An organisation may deliberately prioritise severe need, protect vulnerable people or meet obligations that are not captured by maximising the number of successful interventions. Those priorities belong in the decision policy. They should not be smuggled in through a convenient prediction target.

The practical task is to connect the forecast to an explicit, defensible action rule. Otherwise the organisation has purchased a description of the future and mistaken it for a way to improve that future.

A human in the process is not the missing proof

It is tempting to bridge these gaps by keeping a person responsible for the final decision. That may be sensible, but it does not establish that the combined process will perform better.

A 2024 meta-analysis by Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone covered 106 experiments. On average, human–AI combinations performed worse than the better of the human-only or AI-only alternatives, although they performed better than humans alone. Results varied by task; decision tasks showed different patterns from content-creation tasks. ⁴

These experiments predate current systems and do not establish that every human–AI arrangement fails. They do challenge the assumption that putting the two together automatically combines their strengths. ⁴

For a manager, the implication is to test the actual working arrangement. Does the person understand what the score means? Can they obtain missing information? Do they have time and authority to challenge it? Does the process learn when an override was justified, rather than merely counting overrides as resistance?

Those conditions are part of the decision system. They are not footnotes to the model’s performance report.

The prediction-to-decision test

Before allowing a score to change an operational decision, ask the project owner to complete five statements:

On a narrow screen, each statement is shown as a separate card.

An accuracy figure answers only part of the first statement. A proposal unable to complete the others is not yet a demonstrated improvement in decision-making.

Test the result that management actually needs

The lesson is not that organisations should abandon prediction. It is that model evaluation and decision evaluation answer different questions.

Historical testing can establish whether a model recognises useful patterns. It cannot, on its own, establish the effect of a new intervention, a new allocation rule or a changed relationship between staff and the system.

That calls for proportionate evaluation of the proposed process. Depending on the setting, this may include specialist review, retrospective checks, observing recommendations without acting on them, and carefully governed prospective testing. In consequential settings, the appropriate design must be agreed with the relevant domain and risk specialists; this is not a licence to experiment on people merely to obtain cleaner data.

The change in the procurement conversation is straightforward. Do not ask only for evidence that the forecast is good. Ask for evidence that the organisation’s proposed use of it produces a better result than a realistic alternative.

A model can be right about what happened and still support the wrong next move. The management responsibility begins where the prediction ends.