Skip to article content

Multi-Agent Systems: Architecture and Design Patterns

A multi-agent system splits an AI workload across several LLM agents, each with its own context, tools and role, coordinated by an orchestrator, a handoff protocol or shared state. Use one when a task breaks into independent subtasks or needs strict separation of privileges; otherwise stay with a single agent, because every extra agent multiplies token cost and adds a coordination seam — and the seams are where most multi-agent failures happen.

  • Agentic AI
  • Thundrom Engineering
  • Nov 9, 2026
  • 17 min read
Abstract network of glowing connected nodes, representing AI agents exchanging messages in a multi-agent system
Key takeaways
  • Multi-agent is a cost–quality trade, not an upgrade. Anthropic measured a 90.2% gain over a single agent on its internal research eval — at roughly 15× the tokens of a chat.
  • Seven patterns cover almost everything: orchestrator–workers, pipeline, router/handoff, parallel fan-out, evaluator–optimizer, blackboard and hierarchical teams.
  • The failures are architectural. In the MAST study, specification issues (41.77%) and inter-agent misalignment (36.94%) dominate; better models alone do not fix them.
  • Context is the hard part. Delegate with self-contained contracts and pass artifacts by reference, not lossy prose summaries.
  • Every agent is a security principal. Scope its tools, treat peer messages as untrusted data, and log every action against the agent that took it.

Every team building agents eventually hits the same wall. The single agent that handled the prototype starts to sag under a growing tool list, a context window stuffed with half-relevant documents, and tasks that would obviously go faster if three things happened at once. The instinct is to add more agents. Done well, that instinct produces systems that are faster, more capable and easier to secure. Done badly, it produces a distributed system with all of the failure modes of microservices and none of the determinism.

The pressure to get this right is real. Gartner named multiagent systems one of its top strategic technology trends for 2026, and the same firm has forecast that over 40% of agentic AI projects will be scrapped by the end of 2027 on cost, unclear value and weak risk controls. Architecture is where those three outcomes are decided.

This guide is the architecture layer of our agentic AI series. It assumes you know what an agent is; if not, start with AI agents as employees. It pairs with our deep dives on why AI agents fail in production, context engineering and the Model Context Protocol.

What a Multi-Agent System Actually Is

Four building blocks, and the line between workflows and agents

Strip away the marketing and a multi-agent system is a set of LLM-driven components, each with a narrow job, plus the machinery that makes them cooperate. In practice every design is a combination of four decisions:

  • Agents. Each has a role prompt, a model, a tool allowlist and its own context window. Different agents can use different models — a strong model to plan, cheaper ones to execute.
  • Topology. Who can talk to, delegate to, or hand off to whom: a star around an orchestrator, a line, a tree, or a shared hub.
  • Communication. Direct messages, transfers of control (handoffs), or reads and writes against shared state. This is where context is gained or lost.
  • Control plane. Budgets, termination rules, verification gates, permissions, tracing and human approval. Most production failures trace back to gaps here.

Workflows vs. agents: who decides what happens next?

Anthropic's widely cited Building Effective Agents draws the most useful line in this space. In a workflow, LLMs and tools are orchestrated through predefined code paths. In an agent, the model dynamically directs its own process and tool use. Multi-agent systems sit on both sides of that line: a pipeline of agents is a workflow, while an orchestrator that decides at runtime how many workers to spawn is agentic. The distinction matters because every decision you hand to a model is a decision you must evaluate, trace and bound.

Where MCP and A2A fit

Two open protocols now frame the plumbing. The Model Context Protocol connects an agent to tools and data — it became a founding project of the Linux Foundation's Agentic AI Foundation in December 2025. The Agent2Agent (A2A) protocol, introduced by Google in 2025 and since moved to Linux Foundation governance, connects agents to other agents: capability discovery through agent cards, task delegation and streaming status updates. In August 2026 Axios reported that A2A would join MCP under the same foundation. The practical split: MCP inside every agent, A2A only where an agent crosses an ownership boundary.

Single Agent or Multi-Agent?

The evidence for both sides, and how to decide

The strongest public case for multiple agents comes from Anthropic's write-up of its multi-agent research system. A lead agent running Claude Opus 4 with Claude Sonnet 4 subagents outperformed a single Opus 4 agent by 90.2% on the company's internal research evaluation, and parallel tool calling cut research time by up to 90% on complex queries. The same post is candid about the bill: agents use about 4× the tokens of a chat interaction, and multi-agent systems about 15×.

90.2%
Gain over single-agent Opus 4 on Anthropic's research eval
Anthropic, Jun 2025
15×
Tokens used by multi-agent systems vs. a chat interaction
Anthropic, Jun 2025
80%
Of BrowseComp performance variance explained by token usage alone
Anthropic, Jun 2025

That last number is the key to reading the result honestly. Much of the multi-agent advantage comes from spending more tokens in parallel, separate context windows — breadth that a single context cannot hold. It is a genuine architectural benefit, but it is bought, not free.

Actions carry implicit decisions, and conflicting decisions carry bad results.
— Walden Yan, Cognition — “Don’t Build Multi-Agents”

The counter-argument is just as well founded. Cognition's essay argues that when parallel subagents cannot see each other's full traces, they make unstated choices that clash at merge time — and that for tightly coupled work like writing a program, a single-threaded agent with careful context compression is more reliable. Anthropic's post agrees on the boundaries: multi-agent designs are a poor fit when all agents need the same context, when there are many dependencies between subtasks, and for most coding tasks.

A practical decision test

Both camps are describing the same variable: coupling. Go multi-agent when you can answer yes to at least one of these, and the task value justifies a several-fold cost increase:

  1. Breadth exceeds one context. The information needed genuinely will not fit, or degrades the agent's focus when it does.
  2. Subtasks are independent. They can be completed without seeing each other's intermediate decisions — research threads, document batches, separate systems.
  3. Tool sprawl hurts selection. One agent with dozens of tools picks the wrong one; specialists with five each do not.
  4. Privileges must be separated. The step that reads untrusted email must not be the step that can wire money.
Single-agent versus multi-agent tradeoffs
DimensionSingle agentMulti-agent
ContextOne window holds everything; risk of overflow and distractionEach agent gets a clean, focused window; risk of lost context at handoffs
ParallelismSequential tool calls (parallel calls within a turn at best)Independent subtasks run concurrently, cutting wall-clock time
Token costLower — roughly 4× a chat interaction per AnthropicHigher — roughly 15× a chat interaction per Anthropic
CoherenceEvery decision is made with full historyImplicit decisions can conflict between agents
Least privilegeOne agent holds every tool and credentialTools and secrets can be scoped per role
DebuggingOne trajectory to readDistributed trace across agents, handoffs and shared state

The Seven Core Multi-Agent Design Patterns

Topologies that cover nearly every production system

Framework names differ — supervisors, crews, swarms, graphs — but the underlying topologies are few. The five workflow patterns in Anthropic's guide (prompt chaining, routing, parallelization, orchestrator–workers and evaluator–optimizer) plus two structures common in larger systems give you a complete vocabulary.

Interactive · topology explorer
Seven ways to wire agents together

Select a pattern. Ratings are Thundrom's qualitative engineering assessment (1–5), not benchmark results.

TaskLeadW1W2W3Merge
Orchestrator–workers a.k.a. supervisor, lead agent

A lead agent decomposes the goal at runtime, spawns specialised workers with narrow instructions, and synthesises their results. The decomposition itself is a model decision, not code.

Use it for

Open-ended work whose subtasks you cannot enumerate up front — research, due diligence, broad codebase audits.

Watch out for

The lead becomes a bottleneck and a single point of misunderstanding; vague delegation produces duplicated or conflicting work.

Parallelism
Coordination overhead
Debuggability

1. Orchestrator–workers (supervisor)

The default for open-ended work. A lead agent plans, delegates to workers in parallel, and synthesises. Anthropic's research system adds a detail worth copying: the lead saves its plan to external memory so it survives context truncation, and a separate citation agent attributes claims after the research is done. The make-or-break skill is delegation — each worker needs an objective, an output format, guidance on tools and sources, and clear task boundaries, or workers duplicate each other.

2. Sequential pipeline

The most underrated pattern. When the steps are known, encode them in code and let each agent do one transformation well. Insert programmatic gates between stages — schema validation, required-field checks, confidence thresholds — so an error stops where it starts rather than compounding downstream.

3. Router and handoff

A triage agent classifies the request and transfers control to a specialist. The OpenAI Agents SDK makes handoffs a first-class primitive; LangGraph models them as edges between nodes. Design each handoff to carry a structured summary — who the user is, what was tried, what is being asked — and cap the number of transfers per conversation to prevent ping-pong.

4. Parallel fan-out

Split in code, run concurrently, reduce deterministically. Two variants matter: sectioning, where workers take different slices of the input, and voting, where several agents attempt the same task and you take consensus — useful for high-stakes classifications. Running a guardrail agent in parallel with the main response is a cheap, effective form of sectioning.

5. Evaluator–optimizer

A generator and a critic iterate until the critic passes the output. It works when the criteria are explicit. Ground the evaluator wherever possible — run the tests, validate the schema, check against retrieved facts — because a model judging another model's prose drifts towards approval. Always bound the number of rounds.

6. Blackboard and shared state

Agents coordinate through a shared, versioned store instead of messaging each other — an idea borrowed from classic AI blackboard systems and now common as graph state in LangGraph or as event streams on a broker such as Kafka. It suits asynchronous, long-running processes. Treat the store like any shared database: one owner per field, optimistic concurrency, and an append-only history of every write.

7. Hierarchical teams

Supervisors managing supervisors. It keeps any one agent's tools and context manageable at very large scale, but every layer adds latency, cost and a lossy summarisation hop. Adopt it only after a flat orchestrator has measurably run out of room.

Context, Contracts and Communication

How work moves between agents without losing meaning

Every boundary between agents is a compression step. The orchestrator knows why a subtask exists; the worker sees only what it is told. Most coordination bugs are information that failed to cross that boundary. Three rules keep it intact:

  • Delegate with contracts, not chat. Each subtask carries a self-contained objective, an output schema, a tool allowlist, and step and token budgets. A worker should never need the orchestrator's history to understand its job.
  • Pass artifacts by reference. Workers write full results to storage — files, rows, documents — and return a pointer plus a short structured summary. The synthesiser reads originals rather than a game of telephone.
  • Make shared decisions explicit. If parallel workers must agree on a convention — naming, units, style, an API version — the orchestrator decides it first and puts it in every contract.

Here is that shape as a framework-neutral orchestrator–workers skeleton. The interesting parts are not the model calls; they are the contract, the hard budget, the validation of every worker output, and the refusal to synthesise over a missing piece.

orchestrator.py — illustrative skeleton
import asynciofrom dataclasses import dataclass # make_agent, planner, synthesizer and validate stand in for your framework# (LangGraph, OpenAI Agents SDK, Microsoft Agent Framework, Google ADK, ...). @dataclass(frozen=True)class SubTask:    id: str    objective: str          # self-contained: no "as discussed above"    output_schema: dict     # machine-checkable definition of done    tools: tuple            # least privilege: this worker sees only these    max_steps: int = 12    max_tokens: int = 40_000 class BudgetExceeded(Exception):    pass class Budget:    def __init__(self, tokens):        self.left = tokens    def charge(self, used):        self.left -= used        if self.left < 0:            raise BudgetExceeded()   # hard stop, not a suggestion async def run_worker(task, budget, trace):    agent = make_agent(tools=task.tools, max_steps=task.max_steps)    with trace.span("agent.worker", task_id=task.id):        res = await agent.run(task.objective, schema=task.output_schema)    budget.charge(res.usage.total_tokens)    ok = validate(res.output, task.output_schema)    return {"task": task, "artifact_ref": res.artifact_ref, "ok": ok} async def orchestrate(goal, trace, total_tokens=400_000):    budget = Budget(total_tokens)    plan = await planner.decompose(goal)      # list of SubTask, max 5    results = await asyncio.gather(        *(run_worker(t, budget, trace) for t in plan),        return_exceptions=True,    )    good = [r for r in results if isinstance(r, dict) and r["ok"]]    if len(good) < len(plan):        # never let the synthesizer paper over a missing piece        return escalate_to_human(goal, results)    return await synthesizer.run(goal, artifacts=[r["artifact_ref"] for r in good])

How Multi-Agent Systems Fail

What the MAST study found across seven frameworks

The most rigorous public failure analysis is “Why Do Multi-Agent LLM Systems Fail?” (Cemri et al., NeurIPS 2025). The authors annotated traces from seven popular multi-agent frameworks across more than 200 tasks, reaching an inter-annotator agreement of κ = 0.88, and organised what they found into the MAST taxonomy: 14 failure modes in three categories.

Share of observed failures by MAST category · source: arXiv:2503.13657
Specification issues41.77%

Step repetition 17.14% · disobeying task spec 10.98% · unaware of termination conditions 9.82% · loss of conversation history 3.33%

Inter-agent misalignment36.94%

Reasoning–action mismatch 13.98% · failure to ask for clarification 11.65% · task derailment 7.15% · conversation reset 2.33%

Task verification21.30%

Premature termination 7.82% · no or incomplete verification 6.82% · incorrect verification 6.66%

Read the leading modes as design defects. Step repetition and unawareness of termination conditions are missing control-plane rules. Failure to ask for clarification and reasoning–action mismatch are contract problems at agent boundaries. The verification category is simply the absence of checks. The authors conclude that better base models, while helpful, are insufficient on their own to guarantee reliable multi-agent performance.

Mapping failure modes to fixes

  • Repetition and runaway loops → hard step ceilings, loop detection on repeated tool calls with identical arguments, and explicit done-criteria in every contract.
  • Misalignment at handoffs → structured handoff payloads, a clarification path back to the orchestrator, and shared decisions written into each contract.
  • Premature or missing verification → programmatic checks against world state and schemas after every consequential step, not a model's assurance that it is finished.
  • Silent partial failure → a synthesiser that refuses to merge when inputs are missing, and one trace that spans every agent so the gap is visible.

Security and Governance Across Agents

Every agent is a principal with its own blast radius

Splitting work across agents is also a security decision — potentially your best one. Prompt injection sits at the top of the OWASP Top 10 for LLM applications, and there is still no reliable way to make a model ignore instructions hidden in content it reads. The defence is structural: limit what a compromised agent can do.

Simon Willison's “lethal trifecta” gives the design rule: an agent that combines access to private data, exposure to untrusted content, and the ability to communicate externally can be tricked into exfiltrating that data. Multi-agent architecture lets you make sure no single agent ever holds all three.

  • Per-agent identity and least privilege. Each agent gets its own service identity, short-lived credentials and the minimal tool allowlist for its role — never a shared super-token.
  • Quarantine untrusted input. The agent that reads web pages, emails or uploads has no write tools. It returns structured, validated data, which a privileged agent consumes as data.
  • Treat peer messages as untrusted. An instruction arriving from another agent is not an instruction from your user. Validate it against a schema and the original task.
  • Human gates on irreversible actions. Payments, deletions, outbound messages and permission changes pause for approval, with the full trace attached.
  • Append-only audit trail. Every action records the agent, model version, tool call, inputs and the upstream request that caused it, so incidents can be reconstructed and responsibility assigned.

An Implementation Playbook

From one agent to a governed multi-agent system

The teams that succeed treat multi-agent as an optimisation they earn with evidence, not a starting architecture. This is the sequence we use on client engagements.

  1. Step 01

    Baseline with one agent

    Build the single-agent version first, with the full tool set and a strong model. Capture 30–50 real tasks with known-good outcomes as your evaluation set. This is the bar every multi-agent design has to beat on quality, latency and cost per successful task.

  2. Step 02

    Find the seams

    Read the traces. Split only where the evidence points: context windows overflowing, tool lists too long for reliable selection, subtasks that are provably independent, or permissions that must not live in the same agent.

  3. Step 03

    Pick the simplest topology that fits

    Known steps → pipeline. Separable domains → router. Independent slices → fan-out. Unknowable decomposition → orchestrator–workers. Reach for hierarchy or a blackboard only after a flatter design has measurably failed.

  4. Step 04

    Write contracts, not prompts

    Every delegation gets an objective, an output schema, a tool allowlist, and step and token budgets. Workers return artifacts by reference with a trace ID, not prose summaries of what they think they did.

  5. Step 05

    Instrument and gate

    One trace per task spanning every agent, tool call and handoff. Programmatic checks between stages. Release only when the multi-agent version beats the baseline on the same evaluation set, run repeatedly.

  6. Step 06

    Govern before you scale

    Per-agent identities and credentials, human approval on irreversible actions, an append-only audit log, and a kill switch. Then — and only then — widen the rollout.

Conclusion

Architecture Is the Agent Strategy

Multi-agent systems are not a more advanced kind of AI; they are a distributed-systems answer to three concrete problems — context that will not fit, work that can run in parallel, and privileges that must be separated. When those problems are real, the evidence says the gains are large. When they are not, the same architecture multiplies cost and creates new places to fail.

The durable skills are the unglamorous ones: choosing the simplest topology, writing contracts instead of prompts, putting hard limits in code, verifying against the world rather than the model, and giving every agent an identity you can audit. Get those right and the choice of framework — and of next year's model — becomes a detail.

Frequently Asked Questions

What engineering leaders ask before committing to multi-agent

A multi-agent system (MAS) is an application in which two or more LLM-driven agents — each with its own instructions, context window and tools — cooperate on a goal. What makes it a systemrather than a pile of prompts is the coordination layer: a topology (who talks to whom), a communication mechanism (direct messages, handoffs or shared state), and the controls around it (budgets, verification and permissions). Gartner listed multiagent systems among its top strategic technology trends for 2026.

When the task splits into subtasks that are genuinely independent, when one context window can no longer hold the information needed, when a single agent's tool list has grown too long to select from reliably, or when different steps need different security privileges. If the steps are tightly coupled and every decision depends on the previous one, a single agent with good context engineering is usually more reliable and far cheaper.

They solve different layers. The Model Context Protocol (MCP) standardises how an agent connects to tools and data; the Agent2Agent protocol (A2A) standardises how independent agents — potentially built by different vendors — discover each other and delegate tasks. Inside a single application you rarely need A2A; your framework's own orchestration is enough. A2A earns its place at organisational boundaries, when your agent must call an agent you do not own.

As few as the task demands, scaled to its difficulty. Anthropic's research system encodes this explicitly: roughly one agent with a handful of tool calls for simple fact-finding, two to four subagents for direct comparisons, and more than ten only for complex research. A useful default is one orchestrator and three to five workers, with a hard cap enforced in code rather than left to the model.

The patterns matter more than the framework. LangGraph, the OpenAI Agents SDK, CrewAI, Microsoft Agent Framework and Google's Agent Development Kit can all express orchestrator–workers, routing and pipelines. Choose on operational properties: durable state and resumability, first-class tracing (ideally OpenTelemetry), support for human-in-the-loop interrupts, and how transparent the framework is about the prompts it sends. Anthropic's own guidance is to understand the code underneath whatever abstraction you adopt.

Yes, materially. Anthropic reports that agents use about four times the tokens of a chat interaction and multi-agent systems about fifteen times. That cost is justified only when the task's value is high and the quality or speed gain is measurable. Track cost per successful task rather than cost per token, and enforce token and step budgets per agent.

Assume any agent that reads untrusted content can be hijacked, and contain the blast radius structurally: give each agent its own identity and the narrowest tool allowlist, never combine private-data access, untrusted input and an outbound channel in one agent, treat inter-agent messages as untrusted data rather than instructions, require human approval for irreversible actions, and keep an append-only audit trail linking every action to the agent, tool call and input that caused it.
Work with Thundrom

Designing a multi-agent system? Get the architecture right first.

We help teams decide whether they need multiple agents at all, choose the topology, and ship it with tracing, evaluation and governance built in.

Explore More

Kubernetes-Native AI: Running GPU Workloads at Scale with KAITO, Kueue, and DRA in 2026

May 25, 2026

Kubernetes-Native AI: Running GPU Workloads at Scale with KAITO, Kueue, and DRA in 2026

READ →
Why AI Agents Fail in Production: The Evals & Observability Stack

July 14, 2026

Why AI Agents Fail in Production: The Evals & Observability Stack

READ →
The MCP Revolution: How Model Context Protocol Is Redefining AI Integration in 2026

May 24, 2026

The MCP Revolution: How Model Context Protocol Is Redefining AI Integration in 2026

READ →