AI snake oil sells best to broken institutions

The most revealing question in an AI procurement meeting may be one that nobody asks: what would we have to change if this product did not exist?

The answer might be staffing, a policy, a badly designed process or a target that cannot sensibly be met. Those are difficult matters to resolve. A product offering to remove the problem without disturbing the organisation can be much easier to buy.

That is the demand-side argument in Arvind Narayanan and Sayash Kapoor’s AI Snake Oil. Their section “AI Snake Oil Is Appealing to Broken Institutions” argues that exposing weak products is not enough. Institutions with misguided incentives may keep seeking technological quick fixes, even if the present suppliers disappear.¹

The authors use “snake oil” for AI that does not and cannot work as claimed, while also examining exaggerated promises around genuine capabilities. The criticism is about what is being promised, not a presumption that anything labelled AI is fraudulent.¹

For leaders, this shifts the examination from “Is the vendor’s claim credible?” to a second, less comfortable question: “What makes that claim so attractive to us?”

Broken does not mean stupid

The phrase “broken institution” should not become an insult aimed at overworked staff or organisations with too little money.

A pressured team may understand its difficulties perfectly well. It may lack authority to simplify a process, change the service promise or secure the resources needed to fulfil it. The people making the immediate decision can be competent and conscientious while the surrounding incentives produce poor choices.

The relevant failure is a gap between the organisation’s purpose and what its decision process rewards. In this article, that is an operational description, not a label for every underfunded school, hospital or public body.

Illustrative scenario: The recruitment example is hypothetical, not a reported organisation.

Imagine a recruitment team judged mainly on how quickly it produces a shortlist. The organisation has not improved its job descriptions, agreed what good performance looks like or made enough experienced staff available to assess candidates. An automated ranking product promises to reduce the pile.

This is a hypothetical case, but it illustrates the incentive. A shortlist can arrive faster even though the organisation has not established that it contains better candidates. The purchase may resolve the team’s visible workload problem while leaving the institution’s hiring problem untouched.

That does not establish that every automated screening tool is ineffective. It identifies why evidence of better hiring must matter more than the relief of receiving a ranked list.

The promise can be to avoid changing anything else

Narayanan and Kapoor discuss overwhelmed hiring managers, financially pressured journalism and educators seeking tools to identify AI-generated essays. Their argument is that technological promises become especially appealing when institutions want to preserve existing arrangements while reducing the effort required to maintain them.¹

The educational example is particularly revealing. A detector appears to offer a way to retain the same assignment and marking process while restoring confidence about who wrote the submission. Whether that promise is justified is a separate empirical question. The attraction of preserving the process helps explain why people might want to believe it.

The book describes the limitations of the detection tools and incidents known at the time. That is historical evidence and analysis, not a fresh test of every detector available today.¹

The broader management mechanism is not confined to education. A technology proposal may promise more than a function. It may promise that nobody needs to renegotiate a workload, confront a policy contradiction or accept a less convenient definition of success.

This is an interpretation of the demand mechanism, not a finding about the motives of every buyer. Its value is diagnostic: it tells a leadership team where to look for the unexamined assumptions beneath its enthusiasm.

A documented warning from school security

A later regulatory case gives a concrete example of the kinds of promises organisations can be offered under pressure.

In November 2024, the US Federal Trade Commission alleged that Evolv Technologies had made deceptive claims about its AI-powered security screening, including weapon detection, false alarms and comparisons with traditional metal detectors. The agency’s account described both missed weapons and harmless objects being flagged.²

A federal court entered a stipulated order on 5 December 2024. Among its provisions were restrictions on unsupported claims and opportunities for certain school customers to cancel contracts. The case concerned the evidence behind the product’s claims; the allegations should not be confused with a trial verdict establishing every factual contention.³

Nothing in that record proves that the schools concerned were “broken”, or that their leaders bought the technology to avoid responsibility. Safety is a legitimate and difficult concern. Nor does the case establish that AI cannot assist security screening.

What it does provide is a reason to separate a compelling promise from demonstrated performance. The further institutional question is ours: did the organisation’s purchasing process have enough independence to test the claim when accepting it would provide such an appealing answer to an urgent problem?

That question applies just as much to a confident private company as to a constrained public institution.

The purchase can succeed politically and fail operationally

Consider what happens when a programme’s internal milestone is installation rather than improved outcomes.

The sponsor can point to action. Procurement can point to a completed competition. A supplier can point to a functioning product. A dashboard can point to cases processed. Each participant may have satisfied a local requirement, while the organisation still cannot show that its underlying service is better.

This is a possible failure pattern, not a claim about the Evolv customers. It matters because it explains how weak evidence can survive several apparently responsible stages of approval.

No single participant needs to be dishonest. The process simply fails to give anyone a sufficiently strong reason, or enough authority, to ask whether the promised outcome has been achieved.

AI can make this harder to see when technical language is allowed to substitute for a clear operational account. But the weakness does not originate with AI. Another management fashion or a different software category could serve the same purpose.

That is why replacing the supplier may leave the problem intact. Unless the demand changes, the institution remains receptive to the next product offering the same convenient escape.

The alternative is not to wait for a perfect institution

There is an important limit to the critique. An organisation does not have to complete a grand reform programme before it can make useful improvements.

A narrowly scoped tool might genuinely reduce administration, make information easier to find or remove avoidable duplication. That could free capacity for difficult work rather than conceal the lack of capacity. The organisation should be allowed to pursue that possibility.

The dividing line is whether the proposal makes a testable contribution to the real constraint.

Return to the hypothetical recruitment team. A tool that helps organise applications and reduces clerical effort can be assessed on those terms. It should not quietly acquire the additional claim that it identifies the best people for the role. The organisation can take the administrative benefit while retaining an honest account of what remains unresolved.

Likewise, a service should count the cost of checking outputs and handling exceptions before treating a technology purchase as a staffing substitute. Claimed savings are not usable capacity until somebody can explain where that capacity will come from and what will happen to it.

This is not resistance to innovation. It is a way to make a modest, useful intervention distinguishable from an expansive promise.

Use the answers to challenge the business case, not to assign a spurious readiness score. The audit diagnoses organisational demand; it does not certify the technology.

Make the difficult question part of procurement

A persuasive AI proposition deserves two tests. The first concerns the supplier: can the product do what is claimed, under conditions that resemble ours?

The second concerns the buyer: would that capability address the actual problem, or mostly make it easier to avoid a difficult organisational choice?

A leadership team that asks only the first can purchase a real capability for the wrong reason. A team that asks neither is an unusually receptive audience for snake oil.

The task is not to eliminate hope from procurement. It is to prevent hope from becoming the evidence. A sound AI purchase should make the organisation more capable of fulfilling its purpose, not merely better equipped to explain why it has bought something.