When people talk about the cost of running an AI agent, they point at the model — the GPU bill, the tokens, the accelerator that does the thinking. But instrument a real agentic system, one that plans a task, calls five tools, retries two of them, reads a vector store, reconciles the results, and streams a response, and you find something inconvenient: a large share of wall-clock time and machine cost never touches an accelerator at all. It goes to JSON parsing, HTTP fan-out, queue draining, database reads, cache lookups, and retry loops. That connective tissue runs on CPUs, and unlike the model call, it runs more or less constantly.
That's the lens I use to read Microsoft's Cobalt 200 announcement. The headline pitches it as "fully optimized for modern agentic AI workloads," which sounds like marketing until you notice the chip is a general-purpose Arm CPU, not an accelerator. The claim only makes sense once you accept that the expensive part of agents is increasingly the plumbing, not the inference.
What the chip actually is
Cobalt 200 is Microsoft's second-generation, in-house Arm server CPU. The specifications are dense:
- 132 active cores per SoC, built on Arm's Neoverse V3 compute subsystems — the highest-performance core in Arm's V-series, a real step up from the Neoverse N2 in Cobalt 100.
- A two-chiplet package with 66 active cores per tile, fabricated on TSMC's 3nm (N3P) process.
- 3 MB of L2 cache per core plus a 192 MB L3 system cache, fed by 12 memory channels.
- Per-core voltage and frequency control, so each core boosts independently based on what it's doing rather than the whole socket moving in lockstep.
Microsoft's top-line number is over 50% more performance than Cobalt 100. The per-workload figures are the ones worth staring at, because they map directly onto the agent-orchestration plane: up to 135% better on cloud database workloads, up to 80% on caching, and up to 40% on web serving.
Why those numbers matter for agents
Read that list again as an agent engineer. A caching layer in front of embeddings and tool results. A database holding conversation state, memory, and audit logs. A web-serving tier terminating requests and fanning them out to model endpoints and external APIs. Those three categories are the runtime of an agent framework once you strip out the model call itself. The workloads Cobalt 200 improved most are exactly the ones an orchestration tier leans on hardest.
Core density is the other half of the story. Agentic systems tend to be embarrassingly concurrent but individually lightweight — thousands of in-flight sessions, each mostly waiting on I/O, each holding a little state. A 132-core socket with generous per-core cache and independent frequency scaling is well-suited to that shape: pack many small, bursty request handlers onto one machine, let idle cores drop their voltage, and let a core spike when a session suddenly needs to parse a large tool response. High throughput per watt, not peak single-thread speed, is the metric that governs the bill here.
There's a telling detail in how Microsoft designed it. Rather than tuning against a handful of benchmarks, the team built a digital-twin simulation and modeled 140 benchmarks against 2,800 combinations of SoC and system parameters — core count, cache size, memory speed, rack configuration — and evaluated over 350,000 configuration candidates. That's a chip shaped around a realistic mix of cloud-native workloads, which is precisely the mix an agent's non-GPU tier produces.
Actually targeting Arm
The catch is that none of this is free if your containers are amd64. Arm silicon runs the Arm instruction set; your orchestration services have to be built for linux/arm64. In practice that's a buildx one-liner and a node-pool selector, not a rewrite:
# Build the agent's orchestration image for Arm
docker buildx build --platform linux/arm64 -t myorg/agent-orchestrator:arm64 --push .
# Add an Arm node pool to AKS (current Cobalt-generation Dpsv6 size shown)
az aks nodepool add \
--resource-group agents-rg \
--cluster-name agents-aks \
--name armpool \
--node-vm-size Standard_D4ps_v6 \
--node-count 3 \
--labels workload=orchestration
# Pin the CPU-bound orchestration tier to Arm nodes; leave GPU inference where it is
nodeSelector:
kubernetes.io/arch: arm64
workload: orchestration
The Dpsv6 and Epsv6 families are today's Cobalt 100 VM sizes (a full physical core per vCPU, at 2–8 GiB of RAM per vCPU depending on family); Cobalt 200 will arrive as the next generation slotted into the same general-purpose and memory-optimized shapes. If your images already build multi-arch — and most base images for Go, .NET, Node, Python, and the JVM now ship arm64 variants — moving the orchestration tier is a scheduling decision, not a porting project.
The mental model that helps: keep the GPU nodes for inference, and treat the CPU plane around them as a separate, Arm-friendly tier you can migrate independently. You do not have to move the whole system to benefit.
The honest caveats
A few things to keep straight. Cobalt 200 does not do inference for large models — that's still accelerator territory, and this chip is not competing for it. Any library with hand-tuned x86 intrinsics and no Arm path will need attention before it runs, let alone runs fast. And Microsoft has announced the silicon and its performance envelope, but broad VM availability is a rollout, not a switch you can flip today; treat the per-workload percentages as vendor figures until you've measured your own services on them.
The takeaway
Stop thinking about Arm-versus-x86 as a benchmark contest and start thinking about where your agent's money actually goes. Profile a production agent for an hour and separate accelerator time from everything else. If the "everything else" — the databases, caches, queues, and request handlers — is a meaningful slice of your compute spend, that slice is a candidate for an Arm node pool, and Cobalt 200 is Microsoft's bet that it's a large enough slice to build a CPU around. Build one service for arm64, put it behind a nodeSelector, and let the numbers, not the marketing, decide how far you take it.
Sources: New Azure Cobalt 200 VMs deliver 50% performance improvement (Microsoft Azure Blog) · Microsoft's Azure Cobalt 200 ARM Chip Delivers 50% Performance Boost (InfoQ) · Azure Cobalt processor-based Virtual Machines (Microsoft Learn)