When the model is capable enough, governance becomes the design problem

The race to build more capable AI models is not ending. Stanford’s 2026 AI Index describes technical capability as accelerating rather than plateauing, with rapid gains continuing across coding, scientific reasoning and multimodal tasks.[1]

Yet that is not the same as saying that every organisation remains constrained principally by model intelligence. In a growing class of bounded professional tasks, current systems can already draft, compare, classify, synthesise, retrieve, use tools and produce credible analysis. The more immediate question is whether those abilities can be organised into a process that behaves consistently enough to deserve reliance.

This changes the nature of the application-design problem. Instead of asking only which model is smartest, serious teams increasingly have to ask where judgement belongs, what the system may treat as evidence, which actions it may take, when it must stop, and who remains accountable.

Capability has not plateaued. Its marginal value is becoming uneven

There is an important distinction between the frontier of AI research and the practical frontier inside a particular workflow.

At the research frontier, modest improvements can unlock longer tasks, unfamiliar environments or new forms of tool use. Inside a bounded workflow, however, the jump from inadequate to adequate is often far more valuable than the jump from very good to slightly better. A model that cannot do the work is useless. Once several models can do it, the choice begins to turn on reliability, latency, cost, evidence handling and integration.

OpenAI’s own agent-building guidance reflects this shift. It recommends establishing an evaluation baseline with the strongest suitable model, then testing whether smaller and cheaper models still meet the required accuracy. It also notes that not every task needs the smartest model.[2] The model remains important, but it is one component in an engineered system rather than the entire product.

This is where apparent diminishing returns need careful wording. It would be wrong to claim that AI capability as a whole is reaching a ceiling. A more defensible proposition is that raw capability may be delivering diminishing visible value in some already-solved task classes, unless an improvement crosses a threshold that makes a new workflow reliable. For many organisations, the larger near-term gain may come from using the capability they already have more intelligently.

The engineering guidance already looks like governance

The clearest evidence is not a sudden outbreak of governance slogans. It is the growing prominence of technical controls that govern what an application can do.

OpenAI's agent-building guide treats evaluations, tool risk ratings, layered guardrails, failure thresholds and human intervention for high-risk or irreversible actions as part of application design.[2] Microsoft's Agent Framework documentation describes explicit workflows, session state, middleware, telemetry and human-in-the-loop support as production building blocks.[3]

Anthropic’s engineering account goes further by showing why a superficial human-in-the-loop control can fail. It reports that users approved roughly 93 per cent of permission prompts in one product setting, creating approval fatigue. Its response was not simply to remind users to be more careful, but to strengthen containment, access boundaries, sandboxes and granular tool permissions.[4] The lesson is architectural: a prompt asking the model to behave, or a user repeatedly clicking “allow”, is weaker than a system that limits what the model is able to reach and change.

Microsoft’s current agentic maturity model makes the vocabulary explicit. It describes a progression from reactive governance dependent on individual diligence towards governance that is embedded earlier in design, partially automated, continuously monitored and integrated into operations. It also calls for differentiated controls according to an agent’s purpose, criticality and autonomy.[5]

NIST supplied the broader logic before today’s agent frameworks existed. Its AI Risk Management Framework treats governance as a cross-cutting function infused throughout the AI lifecycle. It links technical design and development to organisational roles, testing, incident handling, human oversight and executive accountability.[6]

A 2026 research preprint now uses the phrase “governance by design” directly. Based on a qualitative enterprise case, its authors argue that governance is implemented through architectural and working arrangements: what a system is allowed to do, which tools and data it can use, how memory is handled and how improvements are introduced.[9] As a preprint, it should be treated as an early research signal rather than settled evidence. Its framing is nevertheless revealing.

Taken together, these sources suggest that the thinking is moving into the mainstream of serious application design, even when it travels under other names: evals, guardrails, orchestration, permissions, observability, containment, checkpoints and escalation.

Governance is where policy becomes system behaviour

Not every design choice is governance. Choosing a clearer heading, a faster model or a more attractive interface may improve the product without allocating authority or controlling risk.

A design decision becomes governance when it determines what the system is authorised to do, what evidence it may rely upon, what must be preserved, when it must stop, who may override it, or who carries responsibility for the result.

That gives governance at least four connected layers:

• Institutional governance sets the organisation’s risk tolerance, responsibilities, policies and acceptable uses.
• Application governance translates those expectations into permissions, evidence rules, routing, handoffs, review gates and bounded actions.
• Runtime governance monitors behaviour, detects failures, supports intervention and preserves the ability to contain, reverse or retire a system.
• Human governance identifies who retains judgement, who may approve exceptions and who remains accountable when the AI-supported process influences a real decision.

The weakness in many governance programmes is not the absence of policy. It is the missing translation between the policy and the application. A policy can say that human oversight is required. Application design decides where that oversight occurs, what information the human receives, whether the intervention is meaningful, and what happens if no one intervenes.

Our own case: we thought we were solving consistency

Disclosure: Andrew’s Intel is itself being built with AI. This vignette describes our own bounded design experience, not independent evidence of market prevalence or a quantified performance study.

Andrew’s Intel is developing an AI-assisted research and publishing system for decision-maker intelligence. The early design problem appeared to be editorial: how could we produce accurate work across multiple stages and obtain consistent results across repeated runs without flattening every article into a template?[10]

The answer was not simply a longer prompt or a more capable model. We divided the work into distinct stages for discovery, routing, source evaluation, article-family selection, briefing, outlining, drafting, revision and final quality control. Each stage has a defined job and a bounded right to change what earlier stages decided.

Several choices that initially looked like prompt architecture now look like governance:

• Structured handoffs preserve the approved signal, maturity judgement, source caveats and reader outcome instead of allowing every later stage to reinterpret them freely.
• A commercially interested-source protocol prevents vendor, consultancy or sponsored material from entering the article as neutral evidence without independent support.
• Research-preview and readiness labels constrain what the system may imply or recommend.
• Return, hold and rejection statuses create stop conditions rather than forcing every candidate through to publication.
• The final publication decision remains human. The system provides decision support; it does not assume editorial responsibility.

Within our limited testing, this architecture has produced a marked improvement in accuracy across stages and consistency across runs. We did not make the underlying model more intelligent. We reduced the number of places where it could spend its intelligence on the wrong job, lose an earlier decision or strengthen a claim without authority.

The vignette matters because it shows that governance is not confined to autonomous agents moving money or altering databases. Even a comparatively low-stakes editorial system needs rules about evidence, authority, transformation and approval if its output is to become dependable. The controls should be proportionate to the risk, but the design logic is the same.

Management may know governance matters without knowing where it lives

Awareness is probably asymmetrical rather than absent. Boards, leaders and managers increasingly know that AI creates questions of risk, ethics, data, accountability and regulation. What appears less mature is the understanding that those concerns have to be translated into the behaviour of applications and workflows.

The UK government's AI adoption survey report 2025, updated in February 2026, illustrates a broader readiness gap. Among surveyed businesses already using AI, 84 per cent reported at least some human input or checking, yet only 54 per cent felt ready to scale. Among surveyed businesses planning to adopt AI, 34 per cent felt ready to implement it. These are self-reported responses from different groups; they are not a measure of governance-by-design.[7]

Those figures do not prove that governance-by-design is the missing variable. The research did not ask managers whether they understood permissions, routing, provenance, checkpoints or rollback. It does show that awareness of human oversight and willingness to use AI do not automatically produce organisational readiness.

Managers often encounter governance through a familiar institutional vocabulary: acceptable-use policies, approved-tool lists, legal review, committees, training and risk registers. Those can all be necessary. They are not sufficient if no one asks how the rules become executable inside the system.

The regulatory context makes that translation more urgent. The European Commission said its AI Office and national authorities would begin enforcing applicable AI Act rules from 2 August 2026, when new transparency duties also started.[8] The Act's obligations remain phased, so that date is not a claim that every high-risk requirement applied at once. The practical response cannot be compliance theatre: policy documents that describe responsible use while applications retain vague permissions, weak auditability and meaningless approval prompts.

The Governance-by-Design Test

Leaders do not need to become agent-framework engineers. They do need a way to test whether an AI application has converted broad governance intentions into operational controls.

An editorial decision aid, not an external standard or a validated maturity score. Apply controls in proportion to the workflow and consequence.

A warning sign is an answer that depends mainly on “the model was told not to”. Prompts remain important, but they are probabilistic instructions. Where the consequence matters, governance needs stronger mechanisms around them.

Trust follows architecture

Trust is sometimes discussed as though it were a communications objective: explain the AI, publish principles and reassure stakeholders. That reverses the relationship.

Justified trust emerges when a system behaves within visible boundaries, exposes uncertainty, supports challenge, retains accountable human authority and improves after failure. Governance creates the conditions under which trust may be earned; it cannot manufacture trust when the underlying process is opaque or unreliable.

This is also why a slightly less capable model inside a well-governed application may be more valuable than a frontier model inside a loose workflow. The organisation is not buying an intelligence score. It is relying on a system to prepare, inform or execute work under real constraints.

An observatory, not a one-off claim

The current evidence is strong enough to justify a baseline article, but not strong enough to declare that governance-by-design is universal practice. The terminology remains fragmented, provider guidance is commercially interested, and the management-awareness gap has not been measured directly.

Andrew's Intel proposes a Governance-by-Design Observatory: a longitudinal editorial research project tracking whether governance is moving from an external policy function into application and workflow architecture. The proposal is a research direction, not a claim that a representative monitoring programme is already operating.

Such a project would monitor dated guidance and implementation evidence across model providers, cloud platforms, standards bodies, engineering communities, consultancies, adoption research and organisational-management sources. It would code both explicit governance language and practices that perform governance under other names, including evaluations, permissions, provenance, structured handoffs, escalation, human override, observability, auditability, rollback and incident handling.

The comparison would matter as much as the count. We would look for convergence between engineering and management language—or for developers building controls that managers do not recognise while managers write policies that applications do not execute.

This would be applied editorial research rather than a representative academic census. Any eventual method, source posture and limitations should be visible. A month with no meaningful change would count as a finding rather than an invitation to manufacture a trend. The goal would not be to prove the thesis, but to observe whether the field does.

The next advantage may be the system around the model

Model capability will continue to matter, particularly when a new release crosses a threshold that makes previously unreliable work possible. But many organisations are already underusing capabilities they possess because the surrounding process is poorly designed, weakly governed or difficult to trust.

For those organisations, the next major gain may not come from waiting for a smarter model. It may come from deciding where judgement belongs, what evidence counts, which actions are bounded, when the system must stop and who remains responsible.

That is more than prompt engineering. It is application architecture, operating discipline and governance by design. The application is where policy becomes behaviour—and where model capability becomes organisational capability.