Ask a single agent to list the board members of every information-technology company in the S&P 500, and it will do exactly what you'd expect: read the index, pick a company, search, read, record, move to the next one, and repeat until it either finishes or exhausts its context window. It is correct and it is slow, because each lookup blocks the one behind it. The fix is to stop treating the work as a queue and start treating it as a fan-out — one coordinator that splits the list, several workers that each own a slice, and a final step that stitches the pieces back into one answer.
That fan-out is the entire idea behind a parallel agent pipeline, and it is subtler than it looks. Anthropic reported that a lead agent running Claude Opus 4 with Sonnet 4 subagents beat a single Opus 4 agent by 90.2% on their internal research eval — the S&P 500 board-member task is one of their own examples. The number is real, but it hides the actual engineering. The speedup comes almost entirely from work that was independent to begin with, and the cost of getting the split wrong is paid in tokens and tangled state.
The shape of the pipeline
Most production pipelines settle into an orchestrator-worker structure. One lead agent — the orchestrator — receives the task, decides how to divide it, and dispatches subtasks. The workers run at the same time, each with its own context window and its own tools. When they return, the orchestrator collects their outputs and synthesizes a single result.
In Anthropic's system the lead agent spins up 3–5 subagents in parallel, and each subagent fires 3+ tools in parallel of its own, which cut research time by up to 90% on complex queries. The pattern nests: fan-out at the top, fan-out again inside each branch. This is the same structure you'll see described as a supervisor pattern, a dispatcher/collector, or hierarchical orchestration — different names for one coordinator handing independent work to a pool and merging the returns.
The only question that matters: is the work independent?
Parallelism buys you nothing unless the subtasks don't need each other. "Find NVIDIA's board" and "find Oracle's board" are independent — neither waits on the other, and merging them is a concatenation. "Draft the intro based on what the other section concluded" is not independent, and no amount of orchestration makes it so. Before splitting anything, I run one test: if worker B needs a value that worker A produces, they belong on the same agent, not two.
This is also where the bill arrives. Agents already spend about 4× more tokens than a chat interaction; multi-agent systems spend about 15×. On the BrowseComp eval, token usage alone explained roughly 80% of the performance variance, and token usage plus tool-call count plus model choice together explained 95%. That is a blunt but useful rule of thumb:
Multi-agent pipelines earn their cost only when the task genuinely parallelizes and the value of the answer clears a 15× token premium. Below that line, you are paying orchestra rates for a solo.
What the orchestrator actually does
The dispatch loop is almost boring — the leverage is in the decomposition around it.
import asyncio
async def orchestrate(query: str) -> str:
subtasks = decompose(query) # the highest-leverage line here
results = await asyncio.gather(*(
run_subagent(spec) for spec in subtasks
))
return synthesize(query, results) # fan-in is where the work hides
asyncio.gather is trivial. decompose and synthesize are the whole job. A vague subtask description is the single most common way these pipelines go wrong: two workers duplicate each other, or one misreads its slice and quietly returns the wrong thing. Every spec a worker receives should carry four things — an objective, an output format, guidance on which tools and sources to use, and clear task boundaries:
Subtask(
objective="List current board members of NVIDIA",
output_format="JSON array of {name, role, member_since}",
tools=["web_search", "fetch_page"],
boundary="Board directors only — exclude executive officers who don't sit on the board",
)
Spend your prompt-engineering effort here, not on the coordinator's system prompt. The orchestrator's decomposition decisions determine everything the workers do; it is the highest-impact component in the pipeline.
Where these pipelines break
Three failure modes show up again and again, and none of them are in the happy path:
- Synchronous coordination. In the common design the lead agent can't steer a subagent mid-flight, and subagents can't talk to each other. They get their spec, run to completion, and report back. That is fine for independent work and a trap for anything that needs mid-run negotiation — which is another reason the independence test matters before you split.
- Durability. These runs are long, and a worker that dies 40 seconds into a 60-second task shouldn't restart the whole pipeline. Real systems need durable execution, error handling, and state resumption — checkpoint each worker's progress so you can retry the branch, not the batch.
- Fan-in. Merging five partial answers into one coherent result is its own reasoning task. Concatenation works for board members; it fails the moment two workers disagree or overlap, and you need synthesis logic — voting, weighted merge, or an LLM pass — that you designed on purpose.
Gartner data cited across the industry puts roughly 40% of multi-agent pilots failing within six months of production. The pattern behind most of those failures isn't the model — it's shipping a fan-out over work that was never independent, then drowning in tokens and reconciliation bugs.
When to stay single-agent
Not every hard task wants a pipeline. Anthropic is explicit that most coding tasks "involve fewer truly parallelizable tasks than research," and that this architecture is a poor fit for anything requiring all agents to share the same context or carrying many dependencies between agents. Refactoring a module, tracing a bug through a call stack, editing a document where every section depends on the last — these are sequential by nature. One capable agent with a long context beats five arguing about a shared file.
The concrete takeaway: before you reach for an orchestrator, draw your task's dependency graph on paper and count the edges. If the work fans out into branches that never touch, a parallel pipeline can hand you a 90% latency cut and a real quality gain — so spend your budget on sharp subtask specs and a deliberate fan-in step, not on the dispatch call. If the branches keep pointing back at each other, the honest answer is one agent, and you just saved yourself a 15× token bill.
Sources: How we built our multi-agent research system — Anthropic, Multi-Agent Orchestration Patterns: A Practical Guide, 6 Multi-Agent Orchestration Patterns for Production