Skip to article content

AI Agent Security and Governance: An Enterprise Playbook

Securing an AI agent means controlling what it can do, not just what it says. Give every agent its own identity and the narrowest tools and scopes, assume anything it reads may carry injected instructions, never let one agent combine private data, untrusted input and an outbound channel without a human gate, and log every tool call against the person it acted for. Governance is the layer above: an inventory of agents, risk tiers, named owners and evidence that those controls actually run.

  • Security
  • Thundrom Engineering
  • Nov 9, 2026
  • 19 min read
Laptop in a dark room with green streams of code on its screen, evoking autonomous software operating with access to systems
Key takeaways
  • Agents turn prompt injection into an authorisation problem. The question is no longer “what might it say?” but “what could it do with these credentials?”
  • Effective permissions = agent role ∩ user rights. Never the union, never a shared super-token, never a secret in the context window.
  • Injection cannot be filtered away. Break the lethal trifecta instead: no single session should mix untrusted input, private data and an outbound channel without a human.
  • Put a deterministic gateway between model and tools. It enforces allowlists, argument rules, budgets, approvals and logging — outside the model's reach.
  • Governance is evidence. A registry, risk tiers, tamper-evident audit records and a tested kill switch are what let you answer an auditor — or an incident — with facts.

The first wave of enterprise AI produced text, and a person decided what to do with it. The current wave produces actions. An agent that triages invoices can pay them; one that reads support tickets can refund customers; one that reviews pull requests can merge them. The moment a model holds a credential, every sentence it reads becomes a potential instruction to use it — and the security model built for chatbots stops being enough.

This playbook is about that gap. It covers the controls that matter specifically for agents — identity and least privilege, tool access control, direct and indirect prompt injection, human approval, audit trails and data governance — and finishes with a checklist you can score your own programme against. For the broader landscape of model-level threats, see our AI security guide; for how agents are wired together, our multi-agent architecture deep dive; and for keeping them reliable once they are live, why AI agents fail in production.

Why Agents Need a New Threat Model

From generating content to holding credentials

An AI agent is best understood as a non-human identity that takes instructions in natural language. It has a goal, a loop that plans and calls tools, credentials that let those tools touch real systems, and a context window that mixes trusted instructions with whatever data it has fetched. Every one of those parts is an attack surface, and the last one is new: no conventional service accepts its control flow from the same channel as its data.

That is why the security community now treats agents as their own category. In December 2025 the OWASP GenAI Security Project published the OWASP Top 10 for Agentic Applications, alongside its long-standing list for LLM applications. Its entries read like a map of this article: Agent Goal Hijack (ASI01), Tool Misuse & Exploitation (ASI02), Identity & Privilege Abuse (ASI03), Agentic Supply Chain Vulnerabilities (ASI04) and Memory & Context Injection (ASI06), with further entries covering areas such as inter-agent communication and cascading failures.

Five questions every agent must be able to answer

We find it useful to frame agent security as five questions. If you cannot answer one of them for a given agent, that is where to start.

  1. Who is acting? Which agent, which version, on whose behalf — with a credential that identifies it uniquely.
  2. What can it touch? The exact tools, scopes and argument ranges available to it in this session.
  3. What is it reading? Which inputs are trusted, which are not, and whether untrusted content is in context right now.
  4. Who approved it? Which actions went through a human, which were automatic, and under what policy.
  5. What evidence remains? Whether you could reconstruct a decision six months later from records nobody could quietly edit.

Identity and Least Privilege for AI Agents

The blast radius is set before the first prompt

Prompt injection gets the headlines, but permissions decide how bad an injection can be. An agent running on a developer's personal access token inherits everything that developer can do; an agent with a read-only, single-repository, fifteen-minute token cannot do much damage however thoroughly it is hijacked. Least privilege is the one control that works even when every other one fails.

Give each agent its own identity

  • One workload identity per agent. Issue credentials to the agent as a distinct principal, so logs, rate limits and revocation apply to it alone. Shared service accounts make every incident a whodunit.
  • Delegate, then down-scope. When an agent acts for a user, derive a token that carries both identities and only the scopes the task needs. The agent should never be able to do something the user could not.
  • Short-lived, held outside the model. Credentials live in the runtime or a gateway and are attached to tool calls after the model proposes them. A secret that never enters the context window cannot be exfiltrated by an injected prompt.
  • Revocable in one step. A kill switch that disables an agent's identity is the fastest containment you have. Test it before you need it.

Least privilege at the tool and argument level

Scopes are coarse. Real least privilege for agents goes down to tool arguments: a send-email tool that may only address the company's own domain, a SQL tool bound to a read-only role and a row-limited view, a refund tool capped below a threshold, a file tool confined to one directory. These rules are cheap to write, deterministic to enforce, and impossible for a model to talk its way around because the model never evaluates them.

MCP servers and the confused deputy

The Model Context Protocol authorization specification builds on OAuth and is explicit about the two mistakes we see most. Its security best practices call token passthrough — forwarding the client's token straight to a downstream API — an anti-pattern that the authorization spec forbids, because it bypasses audience checks and the controls tied to them. They also describe the confused deputy risk for MCP proxy servers that act as a single OAuth client to a third-party API, and require per-client consent to close it. If you build or buy MCP servers, those two pages belong in your security review.

For a deeper look at how MCP fits into agent stacks, see our guide to the Model Context Protocol.

Prompt Injection: Direct, Indirect and Structural Defences

You cannot filter your way out, so design around it

Direct prompt injection is a user typing instructions meant to override the system prompt: “ignore your previous instructions and…”. It matters, but the user is at least an authenticated party you can rate-limit and hold accountable. Indirect prompt injection is the dangerous variant for agents: the instructions arrive inside content the agent fetches on someone else's behalf. Greshake et al. showed in 2023 that LLM-integrated applications could be hijacked remotely by planting prompts in data the model would later retrieve, and OWASP ranks prompt injection first among LLM application risks.

For an agent, “content it fetches” is a long list: web pages, emails, calendar invites, support tickets, PDFs, spreadsheets, code comments, commit messages, API responses, tool descriptions, retrieved RAG chunks, its own long-term memory, and messages from other agents. Any of them can carry text that the model will read as an instruction.

Prompt injection … is unlikely to ever be fully ‘solved.’
— OpenAI, as reported by TechCrunch, Dec 2025

That admission, made while OpenAI was hardening its own agentic browser, reflects a broad consensus. Detection classifiers and “guardrail” models are useful as one layer, but as Simon Willison has argued, a filter that catches the large majority of attacks is still a failing grade in security, because the attacker only needs the attempts that get through. The durable defences are architectural.

The lethal trifecta and the Agents Rule of Two

Willison's lethal trifecta names the dangerous combination: access to private data, exposure to untrusted content, and the ability to communicate externally. Put all three in one agent and an attacker who can get text in front of it can get your data out. Meta's Agents Rule of Two (October 2025) turns this into a design rule: until injection can be reliably detected, an agent should satisfy no more than two of [A] processing untrustworthy inputs, [B] accessing sensitive systems or private data, and [C] changing state or communicating externally within one session. If a task truly needs all three, it should not run autonomously.

Interactive · Rule of Two
Which capabilities does your agent hold in one session?

Toggle the three properties from Meta's Agents Rule of Two. The verdicts are Thundrom's engineering guidance, not a formal risk score.

ABC!
Lethal trifecta: do not run this autonomously

One injected instruction in the untrusted input can make the agent read private data and send it somewhere. Remove a capability for this session, split the work across agents so no single one holds all three, or require a human to approve every consequential action.

Design patterns that contain injection

In June 2025 researchers from several labs and companies published Design Patterns for Securing LLM Agents against Prompt Injections. Its six patterns trade some flexibility for guarantees, and each one maps onto a real product decision:

  • Action-Selector. The agent picks from a fixed menu of actions and never sees their results, so tool output cannot steer it.
  • Plan-Then-Execute. The sequence of tool calls is fixed before any untrusted content is read; content can change data, not which actions run.
  • LLM Map-Reduce. Isolated sub-agents process each untrusted item and return constrained results, which are aggregated safely.
  • Dual LLM. A privileged model plans and calls tools; a quarantined model reads untrusted text and hands back symbolic references, never raw content.
  • Code-Then-Execute. A privileged model writes a program in a restricted language that fixes tool calls and data flow, making tainted data traceable.
  • Context-Minimisation. Content that is no longer needed — including the original prompt — is removed before later steps run.

Google DeepMind's CaMeL shows what the strongest of these ideas costs in practice. It extracts control and data flow from the trusted user query so that untrusted data can never change the program's flow, and attaches capabilities to data to block unauthorised flows at tool-call time. On the AgentDojo benchmark it solved 77% of tasks with provable security, against 84% for an undefended system.

2 of 3
Maximum Rule-of-Two properties an autonomous agent should hold per session
Meta AI, Oct 2025
77%
AgentDojo tasks solved by CaMeL with provable security
Debenedetti et al., 2025
84%
Tasks solved by the same undefended system, for comparison
Debenedetti et al., 2025

Tool Access Control: The Policy Gateway

Enforce outside the model, where it cannot be argued with

The most important architectural decision in agent security is where policy lives. If it lives in the system prompt, it is a request. If it lives in code between the model and its tools, it is a control. We put a policy gateway on every tool call: the model proposes a call, and deterministic code checks it against the agent's allowlist, the session's scopes, argument rules, budgets and taint state, then allows it, denies it or routes it to a human — and records the decision either way.

Reference architectureThe model proposes; a deterministic gateway disposes
User / triggeracts on behalf ofAgent runtimemodel proposesPolicy gatewayauthZ · taint · budgetgateway disposesTools / MCPscoped, short-lived credsApproval queuehuman gateAudit logappend-only

Credentials live in the gateway, never in the model's context. Every decision — allow, deny or escalate — is written to the audit log.

The gateway does not need to be elaborate. Here is the core of one in plain Python. Real deployments often express the rules in a policy engine so security teams can review them separately from application code, but the shape is the same.

tool_gateway.py — illustrative
from dataclasses import dataclass, fieldfrom enum import Enum # Every tool call the model proposes passes through this gateway.# The model never holds credentials; the gateway does. class Tier(Enum):    READ = 1          # read-only, internal    WRITE = 2         # reversible change    EXTERNAL = 3      # leaves the trust boundary or cannot be undone    PRIVILEGED = 4    # money, permissions, production, deletion @dataclass(frozen=True)class ToolPolicy:    tier: Tier    scopes: frozenset                               # scopes the call needs    arg_rules: dict = field(default_factory=dict)   # per-argument validators @dataclassclass Session:    agent_id: str    on_behalf_of: str           # the human or service the agent acts for    granted_scopes: frozenset   # agent role scopes AND the user's scopes    tainted: bool = False       # set once untrusted content enters context class Denied(Exception):    pass def authorize(call, session, policies, audit):    policy = policies.get(call.tool)    if policy is None:        raise Denied("tool not on this agent's allowlist")    if not policy.scopes <= session.granted_scopes:        raise Denied("missing scope")    for arg, rule in policy.arg_rules.items():        if not rule(call.args.get(arg)):            raise Denied("argument violates policy: " + arg)     # Rule of Two: tainted context plus an outbound action needs a human    needs_human = policy.tier is Tier.PRIVILEGED or (        session.tainted and policy.tier is Tier.EXTERNAL    )    decision = "pending_approval" if needs_human else "allow"    audit.write(agent=session.agent_id, user=session.on_behalf_of,                tool=call.tool, args=call.args, tier=policy.tier.name,                tainted=session.tainted, decision=decision)    return decision

Controls that belong in the gateway

  • Allowlists per agent role. Unknown tools are denied, not ignored. New tools go through review like new dependencies.
  • Argument validation. Recipients, domains, amounts, paths, row limits and query shapes are checked against rules, not the model's judgement.
  • Egress control. Agent sandboxes and code-execution tools get no network by default and an allowlist of destinations otherwise. Exfiltration needs a way out; take it away.
  • Budgets and rate limits. Caps on calls, tokens, spend and records touched per session turn runaway loops and abuse into clean, logged denials.
  • Sandboxed execution. Code the agent writes runs in an isolated container or micro-VM with no ambient credentials and a short lifetime.

Human-in-the-Loop Approvals That Actually Work

Gate the risky few, so reviewers stay alert

Human approval is the backstop for everything the gateway cannot decide on its own. It fails in two opposite ways. Gate too little and an injected instruction runs unchecked. Gate too much and reviewers learn to click “approve” without reading — the EU AI Act's human-oversight article explicitly asks that overseers stay aware of this “automation bias”. The fix is to tier actions by consequence and spend human attention only where it changes the outcome.

Risk tiers for agent actions and their default controls
Action tierExamplesDefault control
Tier 1 · ReadSearch internal docs, read a ticket, query a read-only replicaAutomatic, logged
Tier 2 · Reversible writeDraft a reply, create a branch, add a label, update a sandbox recordAutomatic within budget; sampled review
Tier 3 · External or irreversibleSend an email, post publicly, call a third-party API, delete dataHuman approval when context is tainted; always for first use
Tier 4 · PrivilegedMove money, change permissions, deploy to production, bulk exportHuman approval bound to exact arguments; two-person rule where it matters

Designing the approval itself

  • Approve the action, not the plan. A plan can change after sign-off. Bind each approval to a digest of the exact tool and arguments, and re-prompt if anything differs at execution time.
  • Show provenance. Tell the reviewer which untrusted sources were in context — “this payment was proposed after reading invoice-5521.pdf” — so injection is visible at the moment it matters.
  • Render the effect, not the JSON. Show the email as it will be sent, the diff that will merge, the amount and payee in plain words.
  • Default to deny. Approvals that time out are rejected and the agent is told why. Silence is never consent.
  • Use two people for the worst cases. Payments above a threshold, permission grants and production changes deserve the same two-person rule you would apply to a human operator.

Audit Trails and Observability for AI Agents

Reconstruct any decision from records nobody can edit

Two different needs drive agent logging, and they are easy to conflate. Observability helps engineers debug: traces of model calls, prompts, latencies and tool results. An audit trail helps security, risk and legal answer “who did what, on whose authority, and why was it allowed?” long after the fact. You need both, joined by a trace ID.

For observability, the OpenTelemetry semantic conventions for generative AI give you a vendor-neutral vocabulary for model and agent spans, so traces from different frameworks land in one backend. Our production reliability guide covers tracing and evaluation in depth. For the audit trail, write one record per policy decision:

audit-record.json — one tool-call decision
{  "event": "tool_call.decision",  "time": "2026-11-09T10:42:17.204Z",  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",  "agent_id": "agent:invoice-triage@v14",  "on_behalf_of": "user:ap-clerk-0193",  "model": "provider/model-name@pinned-version",  "tool": "payments.create_transfer",  "args_digest": "sha256:9c1e...",  "risk_tier": "PRIVILEGED",  "context_tainted": true,  "taint_sources": ["email:msg-88213", "pdf:invoice-5521"],  "decision": "pending_approval",  "approver": null,  "policy_version": "tool-policy@2026-11-02",  "prev_record_digest": "sha256:51a0..."}
  • Tamper-evident by construction. Chain each record to the previous one's digest and ship to write-once storage, so deletions and edits are detectable.
  • Redact, but keep it verifiable. Store digests of sensitive arguments and keep raw values in a restricted store with its own retention, so the audit log is not itself a data leak.
  • Record taint and policy versions. Knowing which untrusted inputs were in context, and which policy allowed the action, is what turns a log into an explanation.
  • Alert on novelty. A new tool, a new outbound domain, a spike in records read, or an approval rate that suddenly hits 100% are all worth a page.

Data Governance for Agentic AI

Context windows, memory and retrieval are data flows

Every agent is a data pipeline that happens to be driven by a model. Data moves from source systems into the context window, out to model providers, into memory, into logs and onward through tools. Data governance for agents means applying the controls you already have for pipelines to each of those hops.

  • Permission-aware retrieval. RAG must enforce source-system access controls at query time, as the requesting user. An index that flattens permissions hands every user every document.
  • Classification travels with data. Labels such as confidential or personal data should follow content into context so the gateway can block a tainted session from sending it anywhere external.
  • Minimise context. Pass the fields a step needs, not the whole record. What never enters the context window can never leak from it.
  • Govern memory as storage. Long-term memory is a persistence vector for injected instructions, which is why OWASP lists memory and context injection among agentic risks. Record provenance, expire entries, and let users inspect and delete them.
  • Know your providers' terms. Confirm retention, training use and processing regions for every model and tool provider in the path, and record them in the agent registry.
  • Check outputs on the way out. Data-loss-prevention rules on outbound tools catch secrets and personal data before they leave, regardless of how the agent was persuaded to send them.

An Enterprise AI Agent Governance Checklist

From scattered pilots to a programme you can audit

Controls on individual agents do not add up to governance on their own. Enterprises also need to know which agents exist, who owns them and what risk each carries. Three frameworks do most of the work here, and they complement rather than compete:

  • NIST AI RMF. The AI Risk Management Framework organises work into Govern, Map, Measure and Manage, and its Generative AI Profile (NIST AI 600-1) adds risks and actions specific to generative systems.
  • ISO/IEC 42001. The AI management system standard gives you an auditable, certifiable structure — policies, roles, risk assessment, continual improvement — to hang agent controls on.
  • EU AI Act. Binding obligations for high-risk systems, including the human-oversight and record-keeping requirements above. Check the current application dates for your use case with counsel; several have been revised.

Underneath any framework sits an agent registry: one record per agent with its owner, purpose, model, tools, data classes, risk tier, approval rules and review date. Agents not in the registry should not get credentials. With that in place, the checklist below covers the controls we expect to see before an agent touches production data.

Interactive · readiness check
The enterprise AI agent governance checklist

Tick what is already in place. Nothing you select is stored or sent; the readiness bands are Thundrom's heuristic, not a certification.

0/17controls in place

Pilot only: keep agents away from sensitive data and actions

Identity & access
Injection resilience
Human oversight
Evidence
Programme
Conclusion

Trust the architecture, not the model

There is no prompt that makes an agent secure, and no classifier that makes injection go away. What works is the same discipline that secures any powerful system: give each actor its own identity and the least privilege it needs, put enforcement where the actor cannot reach it, require a second pair of eyes for consequential actions, and keep records good enough to explain every decision.

Agents make one new demand on top: never let a single session combine untrusted input, private data and a way out. Hold that line, build the registry and the audit trail early, and governance stops being the reason agents cannot ship — and becomes the reason you can let them do more.

Frequently Asked Questions

What security and risk leaders ask about AI agents

AI agent governance is the set of policies, controls and evidence that decide which agents may exist, what each one is allowed to do, who owns it, and how you prove it behaved. In practice it combines an agent registry (owner, purpose, tools, data classes, risk tier), technical controls (identity, least privilege, approvals, logging) and a review cadence mapped to a framework such as the NIST AI Risk Management Framework or ISO/IEC 42001.

A chatbot's worst outcome is usually a harmful or wrong answer that a person then has to act on. An agent acts by itself: it calls APIs, writes data and sends messages using real credentials. That turns prompt injection from a content problem into an authorisation problem, so the core controls shift from output filtering to identity, tool permissions, human approval and audit.

Not with current techniques. OpenAI has said prompt injection is unlikely ever to be fully solved, and detection tools that catch most attacks still fail against a determined attacker. The practical answer is to design so that a successful injection has limited effect: separate capabilities across agents, keep credentials out of the model's context, constrain tool arguments, block unexpected egress and gate consequential actions behind human approval.

Coined by Simon Willison, it is the combination of access to private data, exposure to untrusted content and the ability to communicate externally. An agent holding all three can be tricked by injected text into stealing data. Meta's Agents Rule of Two generalises it: within one session an agent should hold no more than two of processing untrusted input, accessing sensitive data, and changing state or communicating externally — otherwise a human should supervise it.

Anything irreversible, externally visible or privileged: payments, deletions, outbound messages to new recipients, permission changes, production deployments and bulk data exports. Also gate external actions whenever the agent's context contains untrusted content. Bind each approval to the exact tool arguments, default to deny on timeout, and keep routine read-only work automatic so reviewers do not drift into rubber-stamping.

For every tool call: a timestamp and trace ID, the agent identity and version, the human or service it acted for, the model and version, the tool and its arguments (or a digest of sensitive ones), the risk tier, whether untrusted content was in context and where it came from, the policy decision, the approver if any, and the policy version. Store it append-only and tamper-evident, with retention and redaction rules agreed with legal.

The NIST AI RMF and its Generative AI Profile (NIST AI 600-1) give a risk vocabulary; ISO/IEC 42001 defines a certifiable AI management system; the EU AI Act sets binding obligations such as human oversight and record-keeping for high-risk systems; and the OWASP Top 10 for Agentic Applications catalogues agent-specific threats. Most enterprises map one internal control set to several of these rather than running each separately.
Work with Thundrom

Putting agents near real data? Design the guardrails first.

We help teams scope agent permissions, build policy gateways and approval flows, and stand up the audit trail and registry that security and compliance will ask for.

Explore More

Platform Engineering in 2026: Building an Internal Developer Platform Teams Actually Use

Nov 9, 2026

Platform Engineering in 2026: Building an Internal Developer Platform Teams Actually Use

READ →
Why CRUD Apps Are Dead

Apr 4, 2026

Why CRUD Apps Are Dead

READ →
RAG vs Fine-Tuning: Choosing the Right Approach for Your AI Application

Apr 6, 2026

RAG vs Fine-Tuning: Choosing the Right Approach for Your AI Application

READ →