The first time you hand an agent a task that takes 40 minutes instead of 40 seconds, the terminal stops being a place you can sit and watch. You kick off a research brief or a financial model, close the laptop, and now you have two questions with no good answer: is it done yet, and is the thing it produced actually correct? Most people wire up the answer to the first question by polling a status endpoint in a loop, and they answer the second question by reading the output themselves. Both habits fall apart the moment you're running ten of these at once.
There are two primitives that replace the babysitting. Webhooks tell you when to look. Outcome grading tells you whether it's any good. They solve different halves of the same problem, and they compose into something better than either alone: a supervision loop that runs without you in it.
Stop polling — push tells you when
A long-running session moves through states: it starts running, it idles waiting for input, it retries after a transient error, it terminates. Polling means asking "are we there yet" on a timer, which is wasteful when nothing has changed and laggy exactly when something has. Webhooks invert it. You register an HTTPS endpoint, subscribe to the state transitions you care about, and the platform posts to you the moment one happens.
The design choice worth copying is that the payload is deliberately thin. A webhook delivers the event type and an id — not the object itself:
{
"type": "event",
"id": "event_01ABC...",
"created_at": "2026-03-18T14:05:22Z",
"data": {
"type": "session.outcome_evaluation_ended",
"id": "sesn_01XYZ..."
}
}
You then GET the resource by that ID. This avoids shipping stale data on a retry and keeps every delivery small — the notification says "something changed here," and you go read the current truth yourself. It's the same reason a good cache invalidation message carries a key, not a value.
Three properties of the delivery model dictate how you write the handler, and skipping any of them will burn you:
- Ordering is not guaranteed.
session.status_idled can arrive before session.outcome_evaluation_ended even when the evaluation finished first. Sort by created_at if sequence matters; never infer state from arrival order.
- Retries repeat the
event.id. Delivery is at-least-once. Treat the handler as idempotent and dedupe on that ID, or you'll double-fire whatever it triggers.
- Anything but a
2xx is a failure, including a 3xx redirect. Roughly 20 consecutive failures auto-disables the endpoint. Acknowledge fast, do the real work off the request thread.
Verification is one line if you use the SDK's unwrap(), which checks the whsec_-prefixed signing secret and rejects any payload older than five minutes — replay protection you'd otherwise forget to build.
Grading tells you whether it's good
Knowing an agent finished is not knowing it succeeded. This is where most homegrown setups quietly fail: the agent declares victory, the run goes green, and nobody notices the citations are dead links until a human opens the doc.
The fix is to not let the agent grade its own homework. You define an outcome — a plain-English description of the task plus a rubric of gradeable criteria — and the platform provisions a separate grader in its own context window. The grader can't see the working agent's reasoning or the shortcuts it took. It sees only the rubric and the artifact, and it has to return a verdict on every criterion before the loop is allowed to continue.
A writer that knows the criteria is still grading its own work. A grader that opens with a fresh context window and nothing but the rubric and the artifact has no choice but to actually do the checks.
That separation is the whole point, and it lines up with what Anthropic's own guidance on evals argues: grade what the agent produced, not the path it took. Process-based tests punish agents for finding a valid route you didn't anticipate; outcome-based grading only asks whether the result holds up.
The grader's verdict drives a revise loop. Each span.outcome_evaluation_end carries a result:
satisfied — every criterion met; the session goes idle.needs_revision — the agent gets the feedback and starts another iteration.max_iterations_reached — it hit the cap (max_iterations defaults to 3, max 20) without passing.failed — the rubric doesn't apply to what was produced.interrupted — you cut it off.
The quality of the whole system rides on the rubric. Vague criteria produce noisy verdicts, so make each line checkable: not "covers demand charges" but "states a $/kW figure or a percent of operating cost." Anticipate the shortcuts and forbid them explicitly — "do not corroborate a quote via mirrors or search snippets; fetch the source URL and match the string verbatim." A grader told exactly what counts as proof will send back a real failure the first pass and a real pass the third, instead of rubber-stamping.
The loop that runs without you
Put the two together and you get a control plane. The agent works; the grader judges; the revise loop turns; and a single webhook tells your system the verdict is in:
@app.post("/webhook")
def webhook():
event = client.beta.webhooks.unwrap(request.get_data(), dict(request.headers))
if event.data.type == "session.outcome_evaluation_ended":
session = client.beta.sessions.retrieve(event.data.id)
result = session.outcome_evaluations[-1].result
if result == "satisfied":
ship(session) # deliverables live in /mnt/session/outputs/
elif result in ("max_iterations_reached", "failed"):
escalate_to_human(session, result)
# needs_revision: the agent is already iterating; do nothing
return "", 204
Notice what a human is now for. Not watching progress bars, not eyeballing every output — only the max_iterations_reached and failed branches, the small fraction of runs that couldn't clear the bar on their own. That's the supervision ratio you actually want: attention spent only where the automated judge couldn't decide.
The takeaway
If you're standing up long-running agents this quarter, wire these two before you wire anything else, and wire them in this order. Write the rubric first — the exercise of stating what "done and correct" means in checkable lines is worth doing even if you never automate the grading, because a task you can't grade is a task you can't safely delegate. Then subscribe to the outcome-ended event so your system, not a person, is the first to learn the verdict. A human reviewing every agent run doesn't scale past a handful. A rubric and a webhook scale to as many as you can pay for.
Sources: Define outcomes — Claude Platform Docs, Subscribe to webhooks — Claude Platform Docs, Outcomes: agents that verify their own work — Claude Cookbook, Demystifying evals for AI agents — Anthropic