The login round-trip returns to a healthy node, the cookie is valid, and the workflow still cannot continue. The missing piece lives in worker memory: tool outputs, approval state, and the checkpoint that said "resume here after auth." The browser kept its side of the bargain. The session design tied continuity to the original process.
Earendil's The Session You Cannot Take With You describes the same failure mode on the inference side. Some APIs keep part of the usable context behind a provider-owned handle, so a transcript on its own does not let another system continue the work. That shape shows up in ordinary web apps too, especially once auth redirects, long-running agent work, or multi-region routing enter the picture.
A session should point to durable, inspectable state. Sticky sessions buy migration time. They belong at the edge of the design, with a removal date.
Portability is the session test
Earendil sets a practical bar for portability. Another system does not need to produce the same next token. It needs enough intelligible state to continue the task without asking the original owner to reopen hidden context.
That maps cleanly to application sessions. You want a design where another healthy instance can:
inspect the stored state for one user or workflow,
load it from durable storage,
replay the next callback or resume step with the same meaning,
explain which stored state led to a tool call or user-visible action.
Earendil's line, "a response ID is not a transcript," lands here too. A cookie key also names server-side state. The important question is where that key resolves. If it resolves to a row in durable storage, any healthy node can continue. If it resolves to memory inside Node 1, continuity depends on that node.
Encryption does not rescue locality on its own. Portability survives when every healthy node in your fleet can validate and decrypt the state it needs. Portability disappears when one node, one region, or one vendor is the only actor that can reopen the blob.
The durable record needs a claim path, not a toy row
Agent-backed flows stretch a request into a process. That process may pause for a tool call, an OAuth redirect, a human approval, or a backoff window. You need a durable record for the pause, and you need a safe way to consume that record once.
A relational database is a good default because you can inspect a row, join it to audit events, and wrap state transitions in a transaction. Postgres fits this model well. Large prompt artifacts and tool outputs can live in object storage, with the database holding references and hashes.
This sketch is illustrative, not a drop-in schema:
create table session_resumes (
resume_id text primary key,
session_id text not null,
kind text not null, -- oauth_callback | agent_resume | human_approval
workflow_id text not null,
checkpoint_id text not null,
expected_generation bigint not null, -- guards the workflow head
subject_id text, -- null before login completes
return_to text not null,
state_nonce_hash text not null,
status text not null, -- pending | consuming | used | expired | cancelled
claimed_by text,
claim_expires_at timestamptz,
used_at timestamptz,
consumed_checkpoint_id text,
created_at timestamptz not null,
expires_at timestamptz not null
);
create table workflow_definitions (
definition_id text not null,
version integer not null,
runtime_ref text not null, -- image tag, module path, or bundle hash
state_schema_version integer not null,
primary key (definition_id, version)
);
create table workflow_checkpoints (
checkpoint_id text primary key,
workflow_id text not null,
generation bigint not null,
definition_id text not null,
definition_version integer not null,
step text not null, -- awaiting_reauth | waiting_tool | ready_to_resume
tool_results_ref text,
tool_results_sha256 text,
model_context_ref text,
approval_state jsonb not null,
retry_count integer not null,
next_action text not null,
created_at timestamptz not null,
unique (workflow_id, generation)
);
create table workflow_heads (
workflow_id text primary key,
checkpoint_id text not null,
generation bigint not null
);
create table workflow_events (
event_id text primary key,
workflow_id text not null,
checkpoint_id text,
actor text not null, -- user:123 | system:callback | worker:node-7
kind text not null, -- resume_claimed | checkpoint_advanced | resume_used
payload jsonb not null,
created_at timestamptz not null
);
The extra fields matter:
expected_generation lets you reject a stale resume after another worker has already advanced the workflow.
claimed_by and claim_expires_at let one handler take temporary ownership of a pending resume without holding a database transaction open across the whole callback.
definition_version pins the checkpoint to the code and state schema that created it.
Without those guards, the row tells you where the workflow used to be. It does not tell you whether another callback already moved it.
Consume the resume token once
The safe pattern has two parts. Claim the resume token once. Advance the workflow with compare-and-swap on the current generation.
Assume the browser returns with:
GET /auth/callback?code=SplxlOBeZQQYbYS6WxSbIA&state=res_01J9Y7C0D4R5N6
Cookie: sid=sess_01J9T0M8P2
Your app already wrote a pending session_resumes row before it redirected to the identity provider. When the callback arrives, the first handler that sees resume_id = state should try to claim it:
update session_resumes
set status = 'consuming',
claimed_by = $request_id,
claim_expires_at = now() + interval '30 seconds'
where resume_id = $resume_id
and status = 'pending'
and expires_at > now()
returning *;
If that update returns one row, this handler owns the resume for the lease window.
If it returns zero rows, the handler should load the row and branch on the current state:
used: return the canonical redirect or the already-computed result.
consuming with an unexpired lease: another node is working. Return a retryable response, or poll workflow state and redirect once the workflow advances.
consuming with an expired lease: try to steal the lease with a compare-and-swap update on claim_expires_at.
expired or cancelled: reject the callback and ask the user to restart the flow.
That deals with concurrent callbacks. Two requests can hit different nodes at the same time. One claims the row. The other sees a concrete state and behaves predictably.
The workflow itself needs the same discipline. After the handler exchanges the auth code and reconstructs the checkpoint state, it should advance the workflow head only if the head still points at the checkpoint named in the resume row:
update workflow_heads
set checkpoint_id = $new_checkpoint_id,
generation = generation + 1
where workflow_id = $workflow_id
and checkpoint_id = $old_checkpoint_id
and generation = $expected_generation;
If that update affects one row, this handler advanced the workflow. If it affects zero rows, another worker moved the head first. The handler should reload workflow_heads and decide whether it is looking at a harmless duplicate or a genuine conflict.
That second guard matters for more than callbacks. Queue deliveries, webhook retries, and manual approvals can all arrive twice. The resume token gates entry. The workflow head guards the state transition.
A few details make this hold up in practice:
Append the new checkpoint and the workflow_heads update in one transaction.
Mark the resume row as used in that same transaction, and record consumed_checkpoint_id.
Put a unique key on any external delivery identifier you receive, such as a webhook event ID.
Give every side effect its own idempotency key if the resumed step can call an external API.
That last point is where teams often overclaim. A durable resume row gives you exactly-once consumption of the token. External side effects still need idempotent handling. If the resumed step can send email, charge a card, or call a provider that does not deduplicate for you, the resume table alone will not save you.
A compact event trail helps operators and future you:
insert into workflow_events (
event_id, workflow_id, checkpoint_id, actor, kind, payload, created_at
) values
($event_id, $workflow_id, $old_checkpoint_id, $actor, 'resume_claimed', $claim_payload, now()),
($event_id_2, $workflow_id, $new_checkpoint_id, $actor, 'checkpoint_advanced', $advance_payload, now()),
($event_id_3, $workflow_id, $new_checkpoint_id, $actor, 'resume_used', $used_payload, now());
You can keep those payloads small. The point is to preserve causality: which resume row claimed which checkpoint, and which checkpoint replaced it.
Pin the workflow version at the checkpoint
Mid-flight code changes create a quieter portability bug. You deploy workflow version 4 while a checkpoint from version 3 still waits on an OAuth callback. The callback arrives, the app loads the durable row, and the runner tries to execute version 4 code against version 3 state.
That can fail in boring ways. A step name changed from awaiting_reauth to awaiting_login_refresh. The approval payload gained a required field. The retry policy moved from code to configuration. You still have the checkpoint. You no longer have code that knows what the checkpoint means.
Store the workflow definition version in every checkpoint and resume against that version on purpose. A practical rule set looks like this:
Keep the old runtime available until no live checkpoints reference it.
If you need to retire old versions sooner, write an explicit migrator that reads a v3 checkpoint, writes a v4 checkpoint, and appends a migration event.
Refuse to run a checkpoint against an unknown definition version. A hard failure beats silent corruption.
The schema above leaves room for that with workflow_definitions and definition_version. A resume handler can load the checkpoint, then load the matching definition record:
select runtime_ref, state_schema_version
from workflow_definitions
where definition_id = $definition_id
and version = $definition_version;
If the definition exists, your runner can dispatch to the right code path. If it does not, you have a concrete operational task: restore that runtime, or run a migration job, or ask the user to restart. That is unpleasant. It is still better than sending version 4 logic into a version 3 checkpoint and hoping the branch names line up.
Affinity gets one job: migration
Sticky sessions still help when you are extracting local state from a legacy app. Turning affinity off while the app still stores auth redirect state, anti-forgery tokens, wizard progress, and agent checkpoints in process memory can create an outage you did not need.
Give affinity a narrow job:
inventory every field that still lives only in worker memory,
write the durable row first on every create or mutation,
teach callbacks and resumes to read durable state before any local fallback,
measure every fallback with a counter you can graph,
drain a node on purpose and watch an active callback land somewhere else.
Once the fallback counter stays at zero on the routes you care about, delete the local path and shorten the affinity TTL. If the load balancer still needs sticky sessions six months later, the setting has turned from migration aid into architecture.
Steal this
Run this test in staging before you argue about regions, caches, or providers:
1. Start an auth-gated workflow on node A.
2. Persist:
- one session_resumes row in status = pending
- one workflow_heads row at generation = 42
- one workflow_checkpoints row for generation = 42
3. Kill or drain node A.
4. Send the auth callback to node B twice in parallel with the same state token.
5. Expect:
- exactly one request claims the resume row
- exactly one new checkpoint appears at generation = 43
- the second request returns a duplicate-safe result
- workflow_events links resume_id -> old checkpoint -> new checkpoint
6. Deploy workflow version 43 while a second callback still waits on generation 42.
7. Resume that second callback and confirm the runner loads the checkpoint's pinned definition version or fails with a version error you recognize.
That test tells you who owns continuity. If the answer changes when you drain the node that started the flow, you still have session state in the wrong place.