Operating model
AI agent governance
AI agent governance is the decision structure around the agents an organisation runs — who is allowed to approve one going live, on what evidence, and what obliges it to stop. Most agent programmes have tooling and policy and no gate, which means the decision is made by whoever deploys. This page sets out four gates, their criteria, and who holds each.
What agent governance has to decide
Governance is often described as a set of principles. Principles do not resolve cases. A governance model earns its name when it can answer four questions with a name and a date attached:
- May this agent exist? Is the use case one we are willing to automate at all?
- May it act without a human? Which of its actions can be act-tier, and under what conditions?
- May it go live? Has it demonstrated, on real traffic, that its constraints hold?
- Must it stop? What condition obliges suspension, and who can invoke it without a meeting?
The fourth is the one almost universally missing. An agent programme with no defined stop condition and no named person able to trigger it has no governance, regardless of how much documentation surrounds it — because the only remaining control is someone noticing and improvising.
The four gates
A lifecycle with explicit entry and exit criteria. The point of gates is not ceremony; it is that each one has a different decision-maker and a different kind of evidence, so an agent cannot progress on enthusiasm alone.
Gate 1 — Proposal
| Decides | Whether this use case should be automated at all |
|---|---|
| Held by | Business owner plus risk function |
| Evidence in | Purpose in one sentence; systems and data it would touch; the actions it would take; who would own it |
| Exit criteria | Named owner; provisional autonomy level; a stated worst credible outcome; agreement the propose-tier version is insufficient |
The last criterion does more work than the rest combined. Would a version that drafts for approval be good enough? If yes, the act-tier version should not proceed, and asking early saves everything downstream.
Gate 2 — Design
| Decides | Whether the constraints are adequate to the consequence |
|---|---|
| Held by | Security or architecture, with the owner |
| Evidence in | Permission matrix rows; identity and credential scope; tool surface including MCP servers; blast radius as rate × detection latency; reversibility; logging plan |
| Exit criteria | Act-tier permissions individually justified; provenance conditions on anything reachable from untrusted input; evidence retention defined; rate caps set |
Gate 3 — Monitored operation
| Decides | Whether observed behaviour matches the design |
|---|---|
| Held by | Owner, with security reviewing the evidence |
| Evidence in | Real traffic with enforcement in monitor mode: what it actually did, what would have been blocked, measured detection latency |
| Exit criteria | Declared tool surface matches observed; no unexplained would-have-blocked events; detection latency within the assumption used at gate 2 |
Gate 3 is the one organisations skip, and it is the only gate that produces evidence rather than opinion. Gates 1 and 2 are judgements about a system that does not exist yet. Running in monitor mode against real traffic is the first point at which anyone finds out whether the assumptions were right — and they routinely are not, usually because detection latency is far worse than assumed.
Gate 4 — Live, with standing conditions
| Decides | What obliges it to stop, and who can stop it |
|---|---|
| Held by | Governance forum |
| Evidence in | Gate 3 results; the stop conditions; the named individuals who may invoke them |
| Exit criteria | Approval recorded with an expiry date; stop conditions published; suspension exercisable by a named person without convening anything |
Approvals expire. An agent approved eighteen months ago, against a model version that has since moved and a tool surface that has since grown, is running on a decision that was made about something else.
Stop conditions worth defining in advance
Written before go-live, when nobody is under pressure and the argument is abstract. Each should be unambiguous enough that invoking it requires no judgement call.
- Actions exceed n per hour, or n per subject
- An action is taken against an object class outside declared scope
- A tool call is observed that no declared tool accounts for
- The tool-definition hash changes for any connected server
- The model version changes without an approved change
- Error or retry rate exceeds a defined threshold
- Any irreversible action is taken outside its stated conditions
- The owner leaves the organisation and no successor is named within n days
- A complaint or incident is attributed to its output
Suspension must be cheap and unglamorous. If stopping an agent requires an incident call and a director, people will argue about whether the threshold was met instead of stopping it. Make it a switch a named person flips, reversible in minutes, with a review afterwards — the asymmetry between a false stop and a missed one justifies it easily.
Who holds what
Agent governance fails in one of two directions: given entirely to engineering, it becomes a design review with no authority to refuse; given entirely to risk, it becomes a queue that teams route around by not calling their system an agent.
| Role | Holds | Explicitly does not hold |
|---|---|---|
| Agent owner (named individual) | Behaviour, stop authority, keeping the inventory record true | Approval of its own act-tier permissions |
| Security / architecture | Gate 2; enforcement design; gate 3 evidence review | Whether the use case is worth doing |
| Risk function | Register maintenance; scoring consistency; escalation | Day-to-day operation |
| Governance forum | Gate 4; residual risk acceptance; expiry dates | Design decisions |
| Legal / DPO | Lawful basis, deployer obligations, transparency duties | Technical constraints |
One line again matters more than the others: an owner cannot approve their own act-tier permissions. That is the entire structural content of agent governance. Everything else is process around it.
Regulatory obligations that attach to the deployer
The common assumption is that AI obligations sit with whoever built the model. Several land on the organisation operating the system.
| Source | Provision | What it asks of an operator |
|---|---|---|
| EU AI Act | Article 26 | Deployers of high-risk systems: use per the provider's instructions, assign human oversight to competent and supported persons, monitor operation, retain logs under your control |
| EU AI Act | Article 50 | Transparency where a person interacts with an AI system, and for certain generated content |
| UK / EU GDPR | Article 22 | Restrictions on decisions based solely on automated processing producing legal or similarly significant effects |
| UK / EU GDPR | Articles 5, 35 | Purpose limitation and minimisation over what enters a prompt; DPIA where processing is likely to be high risk |
| DORA | Articles 28–30 | Where an agent depends on an ICT third party supporting a critical function, the third-party regime applies to it |
Article 26 is the one that most often reshapes an existing governance model, because "assign human oversight to competent persons" is not satisfied by a named owner who has no practical ability to intervene. Oversight that cannot stop the thing is not oversight — which is why the stop condition and the named individual able to invoke it are a compliance artefact as much as an operational one.
Applicability and timing differ by instrument, sector and role. Take advice on which apply to you; use the table to know where to look.
What produces the evidence each gate asks for
Gates are only as good as the evidence presented at them. Three of the four can be run on documents; gate 3 cannot, by design. What each one needs, and where it comes from:
| Gate | Evidence required | Source |
|---|---|---|
| 1 — Proposal | Purpose, systems, actions, owner | A conversation. No tooling involved, and none needed |
| 2 — Design | Permission rows, credential scope, tool surface, blast radius, logging plan | Partly yours; the registry and agent identity supply the tool surface and credential scope rather than having them declared |
| 3 — Monitored operation | What it actually did, what would have been blocked, measured detection latency | Runtime gateway in monitor mode, with the audit ledger. This gate cannot be run without it — that is the whole point of the gate |
| 4 — Live | Gate 3 results, stop conditions, named individuals | Assurance exports it; lifecycle management carries the expiry date |
And the stop conditions listed earlier are only real if something watches for them. Six of the nine — rate thresholds, out-of-scope actions, undeclared tool calls, definition-hash changes, model version changes, irreversible actions outside their conditions — are runtime signals. The other three, including the owner leaving, are yours to wire into HR and access review.
On Article 26 specifically: "human oversight assigned to competent persons" is not satisfied by a named owner with no practical ability to intervene. Approvals and a working suspension switch are what make oversight demonstrable rather than asserted, and compliance exports the record in the form it gets asked for.
If the estate is large enough that gates 1 and 2 would take a quarter just to work through, the 90-day founding pilot does the inventory across one cloud account and one identity provider, with an owner for every agent.
Starting from nothing
If there is currently no agent governance at all, the order below reaches a defensible position in roughly a quarter without stopping anything that is already working.
| Weeks | Do | Because |
|---|---|---|
| 1–2 | Inventory what exists, including vendor-embedded agents. Scope broadly | Every other step needs a denominator |
| 3 | Name an owner for each. Nothing else | Cheapest control available; fixes orphaned agents immediately |
| 4–5 | Classify actions into read / propose / act. Find every act-tier permission | This is your entire risk surface, and it is usually smaller than feared |
| 6–8 | Write stop conditions for act-tier agents and give someone the switch | Bounds consequence before you have finished understanding likelihood |
| 9–12 | Stand up gates 1 and 2 for anything new; run monitor mode on the largest existing agent | Stops the population growing ungoverned while you catch up |
Deliberately, the policy document is not on this list. It is worth writing after weeks 1 to 5, when you know what you actually have — a policy written first describes an estate nobody has looked at, and its main effect is to create the impression that the question has been handled.
Common questions about AI agent governance
What is AI agent governance?
AI agent governance is the decision structure around the AI agents an organisation operates: who may approve an agent going live, on what evidence, which of its actions may happen without a human, and what conditions oblige it to stop. It is distinct from agent development practice, which is how agents are built.
What are the gates in an AI agent lifecycle?
Four: proposal, deciding whether the use case should be automated; design, deciding whether constraints match consequence; monitored operation, where enforcement runs in observe-only mode against real traffic to test the assumptions; and live approval, which records stop conditions and an expiry date.
Who should approve an AI agent going live?
A governance forum, not the agent's owner. The structural rule that makes agent governance real is that an owner cannot approve their own act-tier permissions — permissions allowing the agent to change state without a further human step.
What are AI agent stop conditions?
Pre-agreed, unambiguous conditions that oblige an agent to be suspended: exceeding a rate threshold, acting outside declared scope, a tool call no declared tool accounts for, a changed tool-definition hash or model version, an irreversible action outside stated conditions, or the owner leaving without a successor.
Does the EU AI Act apply to organisations that only use AI?
Article 26 places obligations on deployers of high-risk AI systems — using them in line with the provider's instructions, assigning human oversight to competent persons, monitoring operation and retaining logs. Those duties attach to the organisation operating the system, not only to the provider that built it.
Where should an organisation with no agent governance start?
Inventory everything including vendor-embedded agents, name an owner for each, classify actions into read, propose and act to find the act-tier surface, then write stop conditions for act-tier agents and give a named person the switch. The policy document is worth writing after that, not before.