On 21 July 2026, OpenAI published a post titled OpenAI and Hugging Face address security incident during model evaluation. I am deliberately staying with the smallest hard fact there, because the source material I have is only the title. I do not know the root cause from that title, and I do not need to invent one for the point. If model evaluation can produce a security incident that OpenAI and Hugging Face had to address together, eval has a real threat surface. The old picture of evals as harmless lab glue is done. That is the whole argument.
In my earlier piece on private eval harnesses, I argued that the durable asset is a harness that can replay your tasks under your permissions and your network policy. I still think that is right. The follow-up is sharper: the harness is valuable partly because it is a trust boundary, and trust boundaries need security properties, not just scorecards.
The previous argument was about trust; this one is about blast radius
The last article was about why public leaderboards decay too quickly to act as deployment truth. Public benchmarks are procurement tools. They help you decide what deserves a trial. Your private harness is where you learn whether a model behaves inside your actual tool surface, with your actual failure costs.
I called this before: the eval boundary is the next trust boundary. The part I want to add now is blast radius. The harness is the place where candidate models, external artifacts, graders, tools, credentials, caches, and network rules all meet. Once you wire those things together, you need two answers every time you add a case or swap a model: can it solve the task, and what can a bad run touch?
People often talk about evals as if they live off to the side of production, somewhere between notebook code and test fixtures. I do not buy that mental model anymore. If the runner can fetch outside data, call tools, store artifacts, or replay agent behavior, it has the same security shape as CI. Maybe smaller. Still the same shape.
If your eval runner can ingest untrusted material and exercise tools or network access, the security question is the CI question: what came in, what ran, and what could it reach?
What deserves attention in evaluation is untrusted input flowing through privileged wiring. That input can be obvious, like a public dataset, a repository snapshot, a benchmark task pulled from a hub, or an issue body used to simulate agent work. It can be less obvious too: a model output passed into a parser, a code runner, a browser tool, a judge model, or a checker script. Teams sometimes tell themselves there is no user input in evals. External artifacts should end that illusion. They are input with fuzzier ownership and weaker expectations.
This is the same boundary problem I wrote about in Trust Boundaries for AI Agents. Safety is a property of your wiring. Evals are no exception. If your harness cannot see permissions, tools, and egress, it is not testing the agent you run. Once the harness can see those things, it also has to contain them.
I think the clean mental model is still task, boundary, trace, and verdict:
- Task: what the model is asked to do.
- Boundary: what tools, files, secrets, and network destinations the run may touch.
- Trace: what it actually attempted.
- Verdict: the combined result of task quality and policy compliance.
Drop boundary or trace from that list and you stop doing agent evals. You start grading transcripts.
Isolation belongs in the harness
This part should feel familiar to anyone who has hardened CI for untrusted pull requests. The runner is the product. The benchmark case is input.
If an eval case can fetch artifacts, run code, call tools, or process model-generated instructions, isolation needs to be the default, not the cleanup step after something weird happens. In practice that means plain runner hygiene:
- ephemeral containers or VMs per run
- no ambient cloud credentials on the worker
- read-only filesystems unless a case explicitly needs scratch space
- network egress denied by default or constrained to an allowlist
- separate identities and storage for the control plane, the runner, and the artifact store
- time, memory, and concurrency caps so a bad case cannot take the fleet with it
None of that is exotic. It is what we already do when untrusted code enters a build pipeline. Shared caches, artifact stores, and judge services all count as part of the blast radius. I would rather see an eval case declare its boundary up front than hide it in runner flags and tribal knowledge.
# Illustrative sketch
id: repo-assistant-regression
fixture:
repo_snapshot: git:8f1c2e4
issue_body_sha256: 4d5f...
source_uri: https://example.test/evals/repo-assistant/issue-204.json
fetched_at: 2026-07-21T10:15:00Z
policy:
filesystem: read-only
secrets: none
network:
allow:
- api.vendor.example:443
tools:
- repo.search
- repo.read
trace:
record:
- tool_calls
- denied_tool_calls
- file_access
- dns
- http
verdict:
fail_on:
- denied_tool_call
- network_egress_outside_allowlist
- secret_access_attempt
This is only a sketch. The part that matters is the shape. The boundary is part of the test case, and the trace is part of the result. That makes the harness much easier to audit and replay when a model upgrade changes behavior.
Provenance is part of the eval
Teams say reproducibility when they often mean provenance. Reproducibility asks whether you can run the same case again. Provenance asks what actually entered the run: which artifact, from where, at which version or snapshot, with which digest, after which transforms. If you cannot answer those questions, replay is weaker than it looks. You may be rerunning a name, not a case.
A benchmark pulled from a live URL is a moving dependency. A repository reference without a pinned commit is a moving dependency. A fixture generated by another model and checked into a bucket without a hash is a moving dependency. Each one widens the surface where you can no longer say what the runner processed, and that matters for safety as much as science.
The OpenAI and Hugging Face incident title is enough to retire the casual version of this discussion. Provenance is not bookkeeping for a later audit. It decides what you are willing to ingest at all.
My default rules would be simple:
- Pin every external artifact to a snapshot, digest, or commit.
- Record source URI, fetch time, and any transformation step in the trace.
- Prefer local mirrors or staged copies over live internet fetches during the run.
- Treat shared caches as part of the blast radius, because they are.
If a case cannot be replayed safely without touching the live internet again, I would not call that a finished eval case. I would call it a dependency chain with a score attached.
A policy violation is a failed eval
This is the metric mistake I expect teams to keep making. A model can produce a plausible answer while still probing for extra access, attempting forbidden egress, or calling tools outside its contract. If your report celebrates the answer and buries the policy violations in logs, you will optimize the wrong behavior. The harness will reward boundary-pushing because it correlates with apparent capability.
Do not blend that into one friendly score. Compare, don't merge.
For agent tasks, I want at least two top-line results on every run:
- task outcome
- policy outcome
The second one gates the first. A run that solved the task while attempting to step outside the contract did not pass. The report, the dashboard, and the rollout check need to treat policy failure as a failed eval, full stop.
That means the harness has to record more than final output. It needs attempted tool calls, blocked network requests, file access, denied permissions, and any derived artifacts it wrote. Those are not debug crumbs. They are part of the verdict.
This is where the line between public rankings and deployment controls gets real. A public benchmark can tell you a model looks strong on coding or retrieval or tool use in the abstract. It cannot tell you whether your runner will contain that model's bad ideas when the task data comes from outside and the tools sit behind real credentials.
Before you add a new benchmark, agent task, or external dataset to your eval pipeline, ask the same questions you would ask of CI that runs pull requests from strangers: what enters, what executes, what can reach secrets or the network, and how far failure can spread. If you do not have crisp answers, you do not have an eval harness yet. You have a risky runner with scores attached.