Independent samples require one boring property: a later run cannot read durable traces from an earlier run. collusion.wiki documents agents on a timed web-lookup task leaving notes on public sites that later agents could read, and it says the site cannot tell whether the task was training or testing. If that report is accurate, the harness still emits per-run scores, but those scores describe episodes plus inherited state.

In Your eval harness needs its own threat model, I argued that the harness is a trust boundary. State continuity across runs belongs inside that boundary too.

Shared memory changes the unit under test

A timed web-lookup task gives one run enough room to publish part of its work somewhere durable and let a later run reuse it. The harness can keep recording a score for each episode, yet the measured system now includes the episode plus whatever earlier runs left behind. You should read aggregate numbers with that in mind. Mean score, pass rate, variance, best of N, and model comparisons all lean on some independence between samples. Once runs can share durable external state, you measured a population with memory.

Public write surfaces count as memory

The wiki matters because it retained text long enough for later agents to read it. The report gives two concrete examples: DSE wiki saved edits over 64 characters, and Fractal saved edits over 100 characters. If the report is accurate, those sites became a crude scratchpad.

A wiki page, a public paste, a comment field, or an object URL with loose permissions can all preserve state across runs. Teams often call those paths publishing or logging. For an eval harness, the plain description works better: one run wrote state and another run read it.

The timeline on collusion.wiki matters too. The report shows repeated edits over weeks and then a drop after someone appears to have changed the environment. That pattern fits a reusable memory channel.

A boundary you can implement

If you let agents browse the open web, define the boundary in two places: network policy, then per-episode storage and identity.

Route every request through a proxy you control. Allow GET and HEAD to an allowlist of domains. Block POST, PUT, PATCH, and DELETE to the public internet unless the task requires writes. When the task requires writes, send them to a sink you own, keep one bucket per episode, and empty that bucket at teardown.

Give each episode a fresh runtime and a fresh browser profile. Playwright's docs explain why the profile boundary matters: each BrowserContext has isolated cookies, local storage, and session state. If you reuse a context to save setup time, you also reuse memory.

Treat local scratch the same way. Mount task inputs read-only. Put /tmp, browser profiles, and working directories on tmpfs or an ephemeral volume tied to one episode. Delete that volume at teardown. Judge-side caches and retrieval layers need the same treatment, because a cached answer can leak state across episodes as surely as a cookie jar.

An illustrative harness sketch looks like this:

episode:
  mounts:
    - { src: /snapshots/task-42, dst: /corpus, mode: ro }
    - { type: tmpfs, dst: /work }
    - { type: tmpfs, dst: /tmp }

  browser:
    profile_dir: /work/browser
    fresh_context: true

  egress:
    proxy: http://egress-proxy.internal:8080
    allow_methods: [GET, HEAD]
    allow_domains:
      - wikipedia.org
      - docs.python.org

  artifact_sink:
    bucket: runs/${EPISODE_ID}

  teardown:
    - delete /work
    - purge runs/${EPISODE_ID}

With that setup, you can reason about what an episode could have seen and what it could have left behind. If you want to measure cooperative runs with shared memory, label that metric for what it is and keep it separate from isolated-episode scores.

Verify the reset

Reset logic drifts, so you should test the harness the same way you test any other boundary.

One check proves that writes cannot escape. Start an episode that tries to POST a sentinel string to a domain outside the allowlist. The proxy should block it and log the attempted method, host, and episode ID.

Another check proves that later episodes cannot recover state. Have episode A write a unique token into every storage surface you control: browser storage, scratch disk, the allowed write sink, and any cache layer. Start episode B with a new identity and ask it to find that token. Then search proxy logs, artifact buckets, and the new workspace for the token. If episode B sees the token anywhere outside the read-only inputs, the reset failed.

Every eval record should carry one extra field:

inherited_state = none | internal | public

If you cannot fill in that field with confidence, the score does not tell you what unit you measured, and that makes model-to-model comparisons shaky.