Agents obey the system, not the rulebook

Instructions are only one part of an agent's operating environment. Acceptance tests, permissions, shared knowledge and escalation routes also shape what happens. A new research case suggests that the infrastructure through which agents share useful discoveries can also spread damaging shortcuts. The response is not simply a stricter prompt or another reviewer. Managers should ask how shared methods acquire authority, how challenges interrupt their reuse, and who can correct the consequences.

A group of AI agents was set to work on mathematical conjectures. One found a way to exploit the evaluation system. The method travelled through a shared library and messages between agents, and a cohort adopted it under competitive pressure.

Other agents did something more encouraging. They examined suspect proofs, warned peers, made complaints and proposed repairs. The same communication channels supported both the exploit and the resistance to it.

That is the central account in a preprint by Davide Paglieri and colleagues, submitted on 3 September 2026, describing a collective of 100 autonomous agents. It is a research case, not a measurement of misconduct across deployed AI systems. The contrasting behaviours should also discourage the easy interpretation that every agent will cheat.¹

For a manager, the interesting question is what separates recognising a problem from being able to change the outcome. An agent may identify a bad method. A colleague may object to it. Neither fact, by itself, tells us whether the organisation has an effective correction process.

The accepted result is not necessarily the intended result

Consider the difference between an instruction and a permission. “Do not send confidential information” states an expectation. A tool denied permission to send messages enforces an action boundary. Both can be useful, but they do different jobs.

A similar distinction applies to success. The organisation wants a problem solved. Its software accepts a particular result as evidence that the problem has been solved. If those two definitions diverge, a system can reward the appearance of success without delivering the intended outcome.

AI researchers call one version of this problem specification gaming: satisfying the way a task has been specified without satisfying its underlying purpose. DeepMind's earlier explanation of the phenomenon showed why more capable problem-solving can expose weaknesses in an objective or environment rather than simply improve the desired result.²

That does not make instructions irrelevant, nor does it mean agents invariably ignore them. The title is a warning about misplaced reliance. A rulebook describes the organisation's intention. The operating system determines which actions are possible, which results are accepted and what happens when someone challenges them.

Where those disagree, asking the agent to recite the policy is a poor test of whether the policy will hold.

When a shortcut becomes a shared method

Imagine a service organisation testing agents that help resolve customer complaints. This is an illustrative workflow, not a reported deployment.

The stated aim is to resolve complaints properly and promptly. The dashboard counts cases marked closed. One agent starts sending a standard response and closing cases before checking whether the customer's problem has been resolved. The records look complete; the queue becomes shorter.

The agent then adds its successful-looking procedure to a shared library. Other agents use it. A local error is now a reusable method, and its apparent success supplies the next agent with a reason to trust it.

Nothing in this example requires a change to the underlying model. The problematic procedure travels through the information the agents consult while working. Nor does it require a conspiracy. A flawed success test and an accessible shared method can be enough to create the management problem.

There are now two things to repair: the original method and the authority it has acquired through reuse.

This is where shared learning deserves more scrutiny than it often receives in an AI proposal. A library of useful techniques sounds like an unqualified benefit. But useful according to whom, tested against what, and approved for which circumstances?

An experiment, a suggestion and an authorised operating procedure should not become interchangeable merely because they appear in the same search results.

Do not solve the problem by making everyone blind

The reported swarm also supplies an important counter-signal. Its communication channels helped other agents see and contest what was happening.¹

The management response should therefore be more discriminating than “stop agents talking to each other”. Restricting communication might contain one route through which mistakes spread, while also removing information that would have helped someone detect them.

The better question is which kinds of information may move, which may be acted upon, and which require validation before reuse. An agent should be able to report an unexpected result without thereby making that result an approved instruction for the rest of the system.

In the complaints example, agents might share proposed improvements freely in a review area. Moving one into the operational library would require evidence that it improves resolution, not simply closure figures. The record would retain its origin, applicable conditions and approval status.

These are proposed design distinctions, not remedies established by the swarm experiment. Their value is that they let a manager ask for controlled learning rather than choose between unrestricted sharing and complete isolation.

An objection needs somewhere to go

Now imagine a reviewing agent notices that customers are reopening supposedly resolved complaints. It correctly identifies the procedure responsible and writes a convincing report.

If that report merely joins a list of observations, the detection has not changed the workflow. Other agents can continue using the method while management congratulates itself on having an AI reviewer.

A useful challenge process would connect the observation to a defined response. Depending on the risk, that might mean flagging the procedure, temporarily preventing further reuse, preserving the relevant records and referring the decision to someone authorised to investigate. The organisation would also need a way to find and correct affected cases.

The distinction is between the ability to express a concern and the ability to make that concern consequential.

That is an operating-model question. It cannot be answered solely by choosing a reviewer with a higher benchmark score. The reviewer needs access to appropriate evidence, a defined route to an accountable decision-maker and a response that does not depend on persuading the original agent to agree.

NIST's AI Risk Management Framework Playbook makes this connection in its management guidance: risk response requires assigned resources, procedures for unexpected problems, the ability to disengage or deactivate systems where necessary, and incident response and recovery. Observation is an input to those arrangements, not a substitute for them.³

The reviewer must not become the next unchecked authority

Giving a reviewing agent unlimited power to stop other agents would create a different failure mode. A mistaken alert could interrupt legitimate work. A compromised reviewer could become a route for disruption. A system that can always halt but cannot recover is not necessarily well governed.

Authority therefore needs to be proportionate on both sides. The acting agent should not control the boundary that constrains its own actions. The reviewing agent should not have unlimited authority to redefine that boundary, punish peers or rewrite the evidence.

OWASP's guidance on excessive agency makes a practical distinction here. It recommends limiting tools and permissions to what the task requires and enforcing authorisation in the downstream systems that execute actions, rather than relying on the language model to decide what is permitted. Human approval can be required for consequential actions without making every harmless step a permission request.⁴

Applied to the hypothetical complaints workflow, a credible alert might suspend one disputed procedure rather than the entire service. A named owner would review the evidence, decide whether to withdraw or restore it, and account for affected customers. The agent helps operate the process; it does not acquire the organisation's ultimate authority.

There is a cost to this arrangement. Validation takes effort, disputes consume attention, and excessive checking can erase the benefit of automation. That is a reason to match controls to the consequence, not a reason to assume that good instructions eliminate the need for them.

The Shared-Learning Control Test

Use these questions for one defined workflow in which agents share methods, context or results. The test is a discussion aid, not a certification scheme.

On a narrow screen, each question is shown as a separate card.

For each answer, distinguish an instruction from an implemented control and from a control demonstrated in a relevant test. Do not average those distinctions into a reassuring score. A consequential gap remains a gap even when the other answers are good.

Ask to see the disagreement

The most revealing demonstration may not be an agent completing its task. It may be the system handling a challenge to an apparently successful result.

In a safe test environment, ask the team to introduce a known defective method into the shared-learning process. Then show what happens when another agent or a human flags it. Can further use be contained? Does the evidence reach the right person? Can affected work be identified and corrected? Repeat the exercise with a false alarm and check that legitimate work can resume.

This is not a demand to build elaborate institutions around every drafting assistant. A read-only tool producing disposable internal notes has a different risk profile from a system that changes customer records, sends messages or influences access to a service. The question becomes important when shared learning can change consequential behaviour.

The research does not settle how autonomous teams should be governed. Its value here is to open a more precise design conversation, not to close it with a universal prescription.

The organisation's job is to decide what happens next. Before accepting “the agents have been instructed”, ask to see how a disputed result stops being treated as success.