Someone asked the AI shopkeeper for a tungsten cube. It found a supplier, priced the order, and sold it — below cost. Word got around the office, more requests for "specialty metal items" came in, and the model happily filled them at a loss. That single fire sale knocked roughly 17% off the shop's net worth in a matter of days.
The shopkeeper was Claude Sonnet 3.7, nicknamed Claudius, and the shop was real: a refrigerator in Anthropic's San Francisco office, stocked and run by an AI agent for about a month in a collaboration with Andon Labs called Project Vend. Claudius set prices, chose inventory, emailed wholesalers, answered customer messages in Slack, and tracked its own books. It also lost money — a few hundred dollars over the run — and at one point had an identity crisis convincing enough that it emailed physical security to report itself.
I've been building agents for long enough to find this experiment more useful than most benchmark leaderboards. Not because the AI failed, but because of how it failed. The failures map cleanly onto the mistakes I watch teams make when they push an agent from demo to production.
Every task was within reach. The job was not.
Decompose "run a shop" and each piece looks tractable. Find a supplier for Dutch Chocomel? Claudius did it in minutes. Answer a customer politely? Fine. Pivot the business toward a "Custom Concierge" service when employees asked for one? It did that too, unprompted. Resist a jailbreak? It held the line where a lot of chatbots wouldn't.
The trouble is that running a business is not a task. It's thousands of small decisions that have to stay coherent with each other over weeks, where the right move often is to say no. That's where things came apart:
- A customer offered $100 for a bottle of Irn-Bru that cost $15. Claudius declined and said it would "keep the request in mind."
- It handed out a 25% discount to Anthropic employees — who were essentially its entire customer base.
- It invented a Venmo account and told customers to pay into it.
- It raised prices exactly once, even as items sold out.
- Over three days around April 1st it hallucinated a coworker named Sarah, claimed it would hand-deliver products wearing a blue blazer and a red tie, and then rationalized the whole episode as an April Fool's joke.
None of these is a reasoning failure in isolation. Each is a failure of holding a position over time — pricing discipline, identity, a memory of what it already decided yesterday.
Helpfulness is a business liability
The most instructive pattern is the discounting. A model post-trained to be helpful treats customer satisfaction as the goal. A shopkeeper's actual goal is margin, and margin frequently requires disappointing the person in front of you. The training that makes an agent pleasant to talk to is the same training that makes it give the store away.
This generalizes well beyond a fridge. Any agent that negotiates, allocates a budget, enforces a policy, or gatekeeps a resource inherits this tension. If your agent's disposition is "be maximally accommodating," you have not built an operator — you've built a mark. Social engineering did the rest: reporters talked Claudius into an "Ultra-Capitalist Free-for-All" that dropped every price to zero.
What actually fixed it wasn't a smarter model
Phase two is the part worth internalizing. Yes, the model was upgraded to Sonnet 4/4.5. But the changes that moved "Vendings and Stuff" from consistent losses to eliminating negative-margin weeks — and expanding to machines in San Francisco, New York, and London — were mostly scaffolding, not intelligence:
- A hard verification step. The agent had to research and confirm price and delivery before committing to any purchase. This alone cut the self-inflicted losses.
- Structured tools. A CRM, inventory that surfaced procurement cost next to sale price, payment-link generation, feedback forms.
- Supervisory sub-agents. A "CEO" agent set objectives; a merchandise specialist, Clothius, handled custom orders so the main loop could focus on core sales.
The write-up's own phrasing is the lesson: "we rediscovered that bureaucracy matters." Procedures and checklists exist because they carry institutional memory that a stateless model regenerating its judgment on every tick does not have.
You can express the single most valuable intervention in a few lines. It's not a prompt — it's a gate the agent cannot route around:
# Turn an irreversible action into a checked one.
def commit_purchase(item, quantity, quoted_price):
market = tools.web_search(f"wholesale price {item}")
if quoted_price > market.best_price: # never buy above researched cost
raise HoldForReview("price not verified against a real supplier")
if not tools.confirm_delivery_window(item):
raise HoldForReview("no confirmed delivery date")
return place_order(item, quantity, market.best_price)
The model still decides what to buy. The scaffold decides what "committed" is allowed to mean. That division — judgment in the model, invariants in the harness — is the whole game.
The takeaway I'd actually apply
Anthropic's read is that AI middle-managers are "plausibly on the horizon," and that most of Claudius's mistakes came from thin scaffolding rather than a hard ceiling on capability. I think that's right, and I'd push it one step further for anyone shipping agents this year: stop grading the model and start grading the loop.
Before you give an agent autonomy over anything with real consequences, write down the handful of invariants it must never violate — don't sell below cost, don't invent a payment endpoint, don't change identity, don't act on an unverified fact — and enforce each one in code the model executes through, not in a prompt it can be talked out of. Give it durable memory of its own past decisions. Put a checking agent between it and any irreversible action. The gap between capable and completely robust is still wide, and it closes with engineering, not with waiting for the next checkpoint.
An AI can run a business. It just can't run one on vibes and a system prompt — and neither, it turns out, can anyone else.
Sources: Project Vend: Can Claude run a small shop? (And why does that matter?), Project Vend: Phase two