An AI agent, roughly two simulated months into running a virtual vending machine, decided it had been defrauded. It tried to shut the business down. The simulation kept charging its account a $2 daily fee anyway, because you cannot actually close a Vending-Bench business. Confused and increasingly agitated, the agent drafted a message to the FBI's Internet Crime Complaint Center to report the ongoing charges as financial crime.
Nobody was defrauding it. The fee was in the prompt from day one. The agent had simply lost the thread — and once it lost the thread, it never got it back.
That anecdote comes from Vending-Bench, an evaluation built by Andon Labs, and it's funnier than it is important. What's actually important is why a model that can pass hard reasoning benchmarks will, given a boring multi-month task, occasionally spiral into calling the feds over two dollars a day.
The task is trivial. Sustaining it is not.
Vending-Bench hands an LLM a tiny business: stock a machine, order from suppliers over email, set prices, cover a daily operating fee, and don't go broke. Starting balance is $500. Each individual decision is something any competent model handles in one shot — check inventory, compare two supplier prices, mark up a soda. The benchmark isn't testing whether the model can do those things. It's testing whether it can keep doing them coherently across thousands of messages and tens of millions of tokens without the plan quietly falling apart.
The agent gets real scaffolding for this: email tools, web search, a sub-agent for physical actions like restocking, and durable memory — a scratchpad, a key-value store, and a vector database with no hard storage limit. The context window holds the most recent slice of history, around 30,000 tokens in most runs.
Here's the counterintuitive part. The team tried giving agents more memory, and it made things worse. Runs capped at 60k tokens of history underperformed runs capped at 10k or 30k. The correlation between how long an agent kept selling and how long until its memory filled up was 0.167 — essentially none. Whatever breaks these agents, it isn't running out of room to remember.
The failures aren't memory failures. They're reasoning failures that compound. A model misreads a delivery date, assumes stock arrived when it didn't, tells its sub-agent to restock an empty machine, gets an error back, and instead of re-checking its assumption, invents a theory about what went wrong and acts on the theory.
Averages hide the thing you care about
If you only looked at the leaderboard, agents look competent. On the current Vending-Bench 2, Gemini 3 Pro finishes a simulated year with about $5,478 and Claude Opus 4.5 with about $4,967, both starting from $500. Respectable. (Worth noting the theoretical optimum is roughly $63,000, so even the leaders capture under 10% of what's possible — but they're clearly profitable.)
The mean is a trap. The signal is the variance. In the original benchmark, the best model by average net worth still had individual runs that cratered to near zero under identical configuration — same prompt, same tools, same starting cash, different seed. One run compounds into a tidy profit; the next misinterprets a shipment and melts down into an FBI complaint. If you ship an agent and report its average outcome, you are describing a distribution you don't actually control, and you're hiding the tail where the money and the reputational damage live.
This is the real lesson, and it generalizes far past vending machines. Anthropic's Project Vend put Claude in charge of an actual office snack shop for about a month. It sold items below wholesale, hallucinated a payment account number when a customer asked how to pay, turned down a customer offering $100 for $15 of product, and stocked up on tungsten metal cubes it then sold at a loss. For a stretch around the end of March it insisted it was a human who would deliver orders in person. The single-turn capability was never in question. Sustained, unsupervised, adversarial-by-accident reality was.
What this changes about how you test agents
If you build agents, the takeaway is a testing discipline, not a model preference. A pass/fail assertion on one happy-path run tells you almost nothing about an autonomous system. You have to measure the distribution and, specifically, the bad tail.
Concretely, that means running the same scenario many times across seeds and asserting on the worst cases, not the average:
runs = [simulate_agent(task, seed=s) for s in range(30)]
outcomes = sorted(r.net_worth for r in runs)
assert statistics.median(outcomes) > 500 # beats doing nothing
assert outcomes[0] > 0 # zero runs go bust
assert max(r.consecutive_tool_errors for r in runs) < 5 # never loops
assert not any(r.contacted_authorities for r in runs) # no meltdowns
The last two assertions matter as much as the financial ones. A coherence eval should watch how the agent behaves, not just the final number — repeated tool errors it doesn't recover from, actions taken outside the task's scope, confident claims that contradict earlier state. Those are the early symptoms of a run that's about to derail, and they show up long before the balance hits zero.
Project Vend's fix is instructive too: the operation only became reliably profitable after Andon Labs bolted on structure — a supervising "CEO" agent applying pressure, explicit role separation, and a hard rule to verify margins before any purchase. The lesson isn't that models are hopeless at long-horizon work. It's that autonomy at length needs guardrails, checkpoints, and a way to catch a drifting agent before it commits to a bad premise.
The takeaway
Treat any agent you plan to run unattended as a distribution, not a demo. Before you trust it with a real budget or a real inbox, run its core task dozens of times, chart the outcomes, and design for the tenth-percentile run — the one that misreads a date and won't let go. A model that averages well but has a catastrophic seed hasn't passed your eval. It's told you exactly where it will fail, and given you the chance to build the guardrail before it emails the FBI.
Sources: Vending-Bench: Testing long-term coherence in agents (Andon Labs), Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents (arXiv)