Production Databricks Notes: short technical note

A team ships an agent improvement. The evaluation score moves from 0.81 to 0.84. The change gets approved.

Nobody knows whether 0.03 is a result or a rounding error, because nobody has measured what the same evaluation produces when run twice on unchanged input.

This is the cheapest gap to close in an agent evaluation setup and one of the most consistently skipped. It takes an afternoon.

The procedure

Take your evaluation dataset. Freeze the agent: same code, same model, same prompt, same retrieval index. Run the full evaluation five times.

Record the score each time. Compute the standard deviation.

That number is your noise floor. Any reported improvement smaller than it is unsupported.

xychart-beta
    title "Five runs of an UNCHANGED agent, then the claimed improvement"
    x-axis ["run 1", "run 2", "run 3", "run 4", "run 5", "new version"]
    y-axis "Evaluation score" 0.75 --> 0.90
    bar [0.81, 0.84, 0.79, 0.85, 0.82, 0.84]

Alt text: A bar chart of six evaluation scores. Five runs of an unchanged agent score 0.81, 0.84, 0.79, 0.85 and 0.82. A sixth bar labelled "new version" scores 0.84, which is identical to run 2 and lower than run 4 of the unchanged agent. The claimed improvement sits inside the range the unchanged agent already produces.

The chart is the whole argument. The sixth bar is the shipped improvement. It matches run 2 exactly and comes in below run 4, both of which are the same agent evaluated twice.

Five runs is a working minimum for a usable estimate. More is better and the marginal value drops quickly. If the cost of five runs is prohibitive, your dataset is large enough that you should sample it rather than skip the measurement.

What the number tells you

Under 0.01: you can detect small changes. Likely your dataset is large, your criteria are mostly deterministic, or both.

0.01 to 0.05: normal for LLM judges on a few hundred examples. Design your release process around it: only claim improvements above this band, and treat anything inside it as unchanged.

Above 0.05: your evaluation cannot reliably distinguish adjacent versions of the agent. This is usable as a smoke test and not as a gate. Fixing it means a larger dataset, more deterministic criteria, or judge criteria specified more tightly.

The high-variance case usually traces to underspecified judge prompts. A criterion that a careful human would score consistently should be scoreable by a judge consistently. If it is not, the prompt is doing less work than you think it is.

Three sources of variance, and which one you can fix

Judge sampling. The judge model produces different assessments on repeated calls. Reducible by tightening the criterion, sometimes by lowering judge temperature where that is exposed.

Agent sampling. If you re-run the agent each time rather than scoring fixed outputs, agent non-determinism is in the measurement too. Worth separating: score a frozen set of agent outputs to isolate judge variance, then re-run end to end to get total variance. The two numbers answer different questions.

Dataset size. The standard deviation falls with the square root of the number of examples, which is why doubling the dataset does not halve the noise. Going from 100 to 400 examples halves it.

Measure judge variance and total variance separately. If judge variance dominates, fix the judge prompt. If agent variance dominates, your agent is less stable than your evaluation suggests, which is itself a finding worth reporting.

The second measurement: agreement with humans

Variance tells you whether the judge is consistent. It says nothing about whether it is right. A judge that reliably measures the wrong thing has a beautiful noise floor.

Label two hundred examples by hand against the criterion you care about. Run the judge over the same set. Measure agreement.

Look at the disagreements specifically, because they are informative in a way the aggregate number is not. Two patterns recur:

The judge is lenient on a specific failure mode. Frequently the one where the response is fluent and wrong, which is the failure mode you most need to catch. Fluency is the easiest signal for a model to pick up and the least correlated with correctness.

The judge is strict on an acceptable variation. Usually formatting or phrasing that your criterion did not intend to constrain. This is cheaper to fix: tighten the prompt to say what you do not care about.

Re-run this whenever you change the judge model. A judge model update changes your measurement instrument without changing your system, and nothing in the tooling will tell you it happened.

Pin the judge model

By default a built-in judge uses a Databricks-hosted model designed for quality assessments, and the model can be changed with the model argument in the judge definition.

Set it explicitly. An unpinned judge means your historical scores were produced by an instrument that may have changed underneath them, and comparisons across that boundary are invalid in a way that leaves no trace.

Treat changing the judge model as a versioned event with a re-baselining step. Record the judge model version in your release evidence alongside the dataset version and the score. Without both, a score is a number without a unit.

Correlated blindness

If the judge and the agent share a base model, they may share a blind spot. The judge then systematically approves exactly the errors the agent systematically makes.

This is hard to detect from the inside, because the evaluation looks healthy. The human-agreement measurement is what surfaces it, which is another reason that step is not optional.

Using a different model family for judging than for generation costs some inconvenience and removes the whole class of problem. Worth it for anything you are gating releases on.

What this changes in practice

Three sentences in your release process:

Report the noise floor next to every evaluation result.

Do not approve a change whose measured improvement is inside it.

Record the dataset version and the judge model version with the score.

The first one is the one that changes behaviour, because it makes the question visible at the moment someone is deciding whether 0.03 is worth shipping.

Companion to this week's investigation into what an agent trace can and cannot prove, and to the Week 5 article on evaluating AI agents on Databricks. Judge configuration verified against Databricks documentation on 2026-07-29. The specific numeric bands above are my working heuristics from practice rather than a documented standard.