The AI works. That may be the problem

A facial-recognition system wrongly identifying a visitor is an obvious problem. A system correctly identifying someone whom an organisation wants to exclude raises a different question: should the organisation be doing that at all?

The controversy around Madison Square Garden’s use of facial recognition illustrates the distinction. In January 2023, New York’s attorney general sought information about reports that the company was using the technology to exclude lawyers associated with litigation against it. The concern extended beyond misidentification to the potential consequences for access to legal representation.¹

The company defended its policy. In its public statement that month, it explicitly acknowledged using facial recognition to keep adverse attorneys out of its venues, described the exclusion as temporary while litigation continued, and maintained that its practices were lawful.²

There was a dispute about the policy, not merely about whether the software could recognise a face. This article is not a legal determination of that dispute, nor a claim that the system never made mistakes.

The management point is simpler. Correcting a false match would not resolve the objection to a correct one.

“Does it work?” and “Is this acceptable?” are separate questions

Arvind Narayanan and Sayash Kapoor make this distinction early in AI Snake Oil. Alongside systems that fail to deliver their advertised capability, they identify systems that can cause harm while working as intended. Their example includes facial recognition used for mass surveillance.³

That matters because “snake oil” is not a useful label for every objectionable application of AI. A capability may be real, a supplier’s description broadly accurate, and the organisation’s intended use still unacceptable or disproportionate.

The remedies are different. A false claim calls for evidence and challenge. An unreliable system may require technical improvement, restrictions or withdrawal. A harmful use of a reliable system may require changing the purpose, narrowing the permission or deciding not to use it in that way.

A procurement process devoted only to performance can miss that final category. It may ask whether the technology delivers the intended result without examining whether that result deserves approval.

Better performance can strengthen a policy you should question

Return to the venue example. Suppose, for the sake of argument, that every identification is correct. Nobody outside the intended group is stopped. Every person on the exclusion list is recognised.

The argument over exclusion remains. Indeed, the policy can now be enforced more consistently. Anyone challenging it cannot rely on the comforting prospect that imperfect technology will limit its reach.

That is not a reason to prefer faulty facial recognition. Errors create additional harms. It is a reason to put limits on the policy rather than expect defects in the technology to provide the limits accidentally.

The same reasoning applies wherever a capability increases an organisation’s ability to classify, monitor, persuade, restrict or replace. Technical competence makes the intended activity easier. Whether that is good depends on the activity, its scale, its justification and the interests of those affected.

The word “misuse” is not always sufficient either. It can suggest that an otherwise sound system has been diverted from its authorised purpose. Sometimes the problem is the authorised purpose itself. The system is doing exactly what the organisation commissioned it to do.

A legitimate objective still needs limits

Consider a hypothetical professional-services firm introducing detailed monitoring of work. Its stated aim is visibility: managers want a reliable account of activity and workflow rather than relying on impression or incomplete records.

Assume the system delivers that visibility accurately. It does not invent events or falsely attribute them to the wrong employee. That would answer an important technical question, but not settle how much visibility is appropriate.

Should every action be recorded? Who can inspect the history? Can records created to diagnose a workflow later be used for an unrelated disciplinary purpose? How long should they remain available? Could people reasonably object to particular forms of monitoring even when the records are correct?

Those questions do not depend on showing that the model is inaccurate. Nor do they require a claim that employees are secretly less productive or that management has malicious intentions. They concern the power created by successful observation.

The firm might conclude that a limited, transparent system is justified. It might restrict monitoring to agreed processes, prohibit individual ranking or delete detailed records after a short period. The appropriate answer depends on the setting and relevant specialist advice.

What it should not conclude is that passing an accuracy test has answered all the other questions.

Successful substitution can also create losses

Not all harms from successful operation involve surveillance or exclusion.

AI Snake Oil discusses image generation as a mixed case: it can produce useful stock-style images while creating costs for the people whose work is displaced or used in developing the technology. The authors distinguish the reality of the capability from the distribution of its benefits and burdens.⁴

For an organisation buying illustration, the immediate benefit may be real. It might obtain an acceptable image faster or at a lower cost. That does not tell us that everyone affected benefits, or that questions about sourcing, permission and compensation have disappeared.

This is not a claim that every generated image replaces a paid commission, that every training practice is unlawful, or that preserving every existing job should take priority over technological change. Those stronger claims would require their own evidence and argument.

It is a reminder that a successful transaction is not a complete account of its consequences. A cost saving for the purchaser and a loss for another participant can exist at the same time. A serious adoption decision should be able to acknowledge both without pretending either cancels the other automatically.

Do not confuse this with predicting the wrong thing

There is a useful boundary here.

A system that predicts spending when the organisation needs to identify unmet need may be accurate about the wrong measure. That is a mismatch between the prediction and the intended service outcome.

The question in this article is different: suppose the system performs the intended task correctly. Are the purpose, permissions, scale and consequences acceptable?

A monitoring system could produce an entirely faithful activity record, and an exclusion system could find exactly the people its owner intends to exclude. Their successful operation would not resolve a disagreement about whether those practices should exist in that form.

That distinction keeps the response practical. Improving the model cannot settle a disagreement about legitimate authority. The organisation has to make and defend that decision itself.

The success-case review

Before a pilot is expanded, ask the approval group to assume that the advertised capability works reliably within its stated scope. Then examine five questions:

On a narrow screen, each question is shown as a separate card.

This review complements reliability testing; it does not replace it. It is a management and governance prompt, not a substitute for legal or other specialist assessment.

Success is the beginning of another decision

The most convincing demonstration can make a proposal feel as though it has crossed the final hurdle. The technology has done what was asked. The people sponsoring it have evidence rather than a speculative promise.

That is a reason to take the capability seriously. It is also the point at which its consequences deserve closer attention. A weak system may fail to achieve its purpose. A strong one may achieve it at a scale the organisation has not properly considered.

Leaders do not need to become reflexively suspicious of useful AI. They need to insist that usefulness has a beneficiary, a purpose and a boundary.

The final approval question is therefore not simply “Can it do this reliably?” It is: “Given that it can, on what terms should we allow it to?”