Traces, scorers, built-in judges, and what to do when the judge is the thing you cannot trust

Production Databricks Notes: architecture decisions, investigations, implementation patterns, and lessons for data and AI systems that must operate beyond the demo.

Every engineering discipline in the previous four weeks assumed a system whose correctness you can assert. A row is duplicated or it is not. An operation reached the platform or it did not.

Agent systems break that assumption. The output is text, the correct output is a set rather than a value, and the thing most teams reach for to judge it is another model with the same failure modes as the one being judged.

This is a solvable problem. It is not solvable by writing assertions, and the shape of the solution is different enough from ordinary testing that it is worth setting out explicitly.

Three things that get called "evaluation"

Separating these first, because conflating them produces evaluation suites that measure the wrong thing.

Offline evaluation runs a fixed dataset through the agent and scores the results. It answers whether this version is better than the last one. It is a gate before deployment.

Online monitoring scores production traffic as it happens. It answers whether the system is behaving now. It is an alarm, not a gate.

Regression detection compares the two over time. It answers whether something drifted, which is the failure mode agent systems have that deterministic systems mostly do not, because the model, the retrieval corpus, and the user population all move independently.

You need all three eventually. Start with offline, because online monitoring without a baseline tells you a number without telling you whether it is bad.

flowchart LR
    TR["Agent traces<br/>input, retrieval,<br/>tool calls, output"]

    subgraph OFF["OFFLINE: a gate, before deploy"]
        direction TB
        DS["Versioned dataset<br/>representative + known failures<br/>+ adversarial + boundary"]
        DS --> EV["mlflow.genai.evaluate()"]
        EV --> SC1["code scorers, deterministic<br/>+ ground-truth judges<br/>Correctness, RetrievalSufficiency"]
        SC1 --> NF{"Above the<br/>noise floor?"}
        NF -->|No| REJ["Not a result"]
        NF -->|Yes| SHIP["Ship"]
    end

    subgraph ON["ONLINE: an alarm, after deploy"]
        direction TB
        PT["Sampled production traffic"]
        PT --> SC2["no-ground-truth judges only<br/>RetrievalGroundedness, Safety,<br/>UserFrustration"]
        SC2 --> AL["Alert"]
    end

    TR --> OFF
    TR --> ON
    AL -.->|"each new failure becomes<br/>a permanent test case"| DS

Alt text: Agent traces feed two separate paths. The offline path, a gate before deploy, runs a versioned dataset (representative, known failures, adversarial, boundary) through mlflow.genai.evaluate() with deterministic code scorers plus ground-truth judges such as Correctness and RetrievalSufficiency, then asks whether the improvement is above the noise floor: no means not a result, yes means ship. The online path, an alarm after deploy, runs sampled production traffic through no-ground-truth judges only (RetrievalGroundedness, Safety, UserFrustration) to an alert. Each new production failure feeds back into the offline dataset as a permanent test case.

Traces are the unit of evaluation

The first structural decision is what you evaluate. The intuitive answer is the final response. It is the wrong one for anything with more than one step.

An agent that retrieves the wrong documents and then produces a plausible answer from them scores well on response quality and is broken. An agent that retrieves correctly and reasons badly scores identically. Evaluating the output alone cannot distinguish them, so it cannot tell you what to fix.

A trace records the full execution: inputs, retrieval, tool calls, intermediate reasoning, and final output. Evaluating traces lets you attribute a failure to a stage.

Two operational facts about tracing on Databricks shape how you can use it. Experiments not in Unity Catalog are capped at 100,000 traces total, and the documented remedy is to move the experiment into Unity Catalog, which the docs describe as giving long-term retention of large trace volumes with no per-experiment limit. Storage location is a capacity decision, so make it before you accumulate traces rather than after.

Production monitoring is in Beta at the time of writing: "This feature is in Beta. Workspace admins can control access to this feature from the Previews page." It is usable and it is the right mechanism; it is not something to write into a contractual availability commitment yet.

Trace-based monitoring is also a near-real-time signal rather than a real-time one, because scoring runs against sampled traffic on a schedule instead of in the request path. Measure the lag you actually get in your workspace before designing an alert around it, and if you need sub-minute detection, use a different mechanism.

Scorers

A scorer is the unified interface for defining evaluation criteria. Two kinds, and the distinction is the most important one in this article.

Code-based scorers are deterministic functions. They are cheap, fast, and exactly reproducible.

LLM judges use a model to assess quality. They handle criteria no assertion can express, and they cost money, take time, and are themselves non-deterministic.

The rule I would follow: every criterion that can be checked in code should be checked in code. Judges are for what requires judgment. This is not a cost argument, though it is also a cost argument. A deterministic check that fails tells you something is wrong. A judge that scores 0.7 tells you a model's opinion, and you now have two systems to debug instead of one.

A code-based scorer:

from mlflow.genai.scorers import scorer
from mlflow.entities import Feedback

@scorer
def exact_match(outputs, expectations):
    return outputs == expectations["expected_response"]

The documentation's own example annotates these arguments, and that is fine for a scorer you only ever run offline. If you intend to register the scorer for production monitoring, drop the annotations: type hints requiring imports cause serialisation failures. Two further constraints apply to registered scorers, both easy to hit late: they must be defined and registered from a Databricks notebook, and all imports must happen inline inside the function body.

Scorers can return a primitive (bool, int, float, str) or a Feedback object when you want an explicit name and a rationale attached to the result. Return Feedback for anything a human will read later; the rationale is what makes a failing score actionable instead of alarming.

Things that belong in code and frequently end up in a judge: response contains a required citation, output parses as valid JSON, no PII in the response, latency under threshold, the tool call sequence matches an expected shape, no reference to a document outside the permitted set.

Built-in judges

Databricks provides research-validated built-in judges. The table below is from the current documentation and is worth reading closely, because the ground-truth column determines what you can run in production.

Single-turn:

Judge	Requires ground truth	Measures
`RelevanceToQuery`	No	Is the response directly relevant to the user's request?
`RetrievalRelevance`	No	Is the retrieved context directly relevant to the request?
`Safety`	No	Is the content free from harmful, offensive, or toxic material?
`RetrievalGroundedness`	No	Is the response grounded in the information provided in the context?
`Correctness`	Yes	Is the response correct compared to the provided ground truth?
`RetrievalSufficiency`	Yes	Does the context provide all information necessary to generate a response?
`Guidelines`	No	Does the response meet specified natural language criteria?
`ExpectationsGuidelines`	Per-example guidelines	Does the response meet per-example criteria?
`ToolCallCorrectness`	Yes	Are the tool calls and arguments correct for the user query?
`ToolCallEfficiency`	No	Are the tool calls efficient without redundancy?

Multi-turn, assessing an entire session without requiring ground truth: ConversationCompleteness, UserFrustration, KnowledgeRetention, ConversationalGuidelines, ConversationalRoleAdherence, ConversationalSafety, ConversationalToolCallEfficiency.

The ground-truth column is the practical dividing line. Judges that need it can only run offline against a labelled dataset. Judges that do not can run against live production traffic. That is why RetrievalGroundedness and Safety are the backbone of most production monitoring setups and Correctness is not.

Registering a scorer for production is a two-step lifecycle, and the sampling rate is where your monitoring bill is decided:

from mlflow.genai.scorers import Safety, ScorerSamplingConfig

safety_judge = Safety().register(name="my_safety_judge")
safety_judge = safety_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.7))

Three constraints worth knowing up front. There is a maximum of 20 scorers per experiment, so the code-first rule above is a budget decision as well as a correctness one. Custom @scorer functions are supported for monitoring but class-based Scorer subclasses are not. And multi-turn judges group traces into conversations using the mlflow.trace.session tag, completing a session after five minutes of inactivity, so a session that a user resumes after a coffee break scores as two conversations rather than one.

RetrievalGroundedness deserves particular attention for RAG systems. It asks whether the response is supported by the retrieved context, which is the closest available proxy for "did this hallucinate", and it needs no labels. If you monitor one thing in production, monitor this.

UserFrustration is the one I would add second, because it correlates with the failures your users actually report and it is invisible in every other metric.

Running an evaluation

results = mlflow.genai.evaluate(
    data=expectations_eval_dataset_list,
    predict_fn=sample_app,
    scorers=[exact_match, Safety()],
)

Three arguments carry the design: the dataset, the function under test, and the list of scorers. Note that built-in judges are instantiated (Safety()) while a decorated code scorer is passed by reference.

By default each judge uses a Databricks-hosted LLM built for quality assessment, and the judge model can be changed with the model argument in the judge definition. Pinning the judge model is worth doing deliberately, for reasons in the next section.

Custom judges

from mlflow.genai import make_judge

my_judge = make_judge(
    name="cites_policy",
    instructions="Does {{ outputs }} cite a policy document for every factual claim it makes "
                 "in response to {{ inputs }}? Answer only from what is present in the response.",
    model="databricks:/<endpoint>",
)

make_judge() creates a judge from natural-language instructions, with template variables giving access to the agent's inputs, outputs, expected outputs, and the complete trace. Trace-based judges are the interesting case: a judge that can see the execution can assess process rather than only result, which is what lets you evaluate whether the agent took a reasonable path to a right answer.

The failure mode with custom judges is writing a prompt that describes the criterion the way you would explain it to a colleague who already shares your context. Judges do not share your context. "Is the response helpful" produces a number that correlates with response length. Specify the criterion the way you would specify it to a contractor who will be paid regardless of whether you like the result.

Where the judge itself is the problem

Your judge is a model. It has the failure modes of a model.

It can be wrong in a correlated way. If the judge and the agent share a base model, they may share a blind spot, and the judge will systematically approve the failure mode you most need to catch. Using a different model family for judging than for generation is worth the inconvenience.

It drifts. If the judge model is updated, your scores change without your system changing. Every historical comparison becomes invalid and you will not be told. Pin the judge model explicitly and treat changing it as a versioned event with a re-baselining step, exactly as you would treat a schema migration.

It is non-deterministic. The same input can score differently. This means a small regression is indistinguishable from noise unless you have measured your noise floor, which almost nobody does. Run your judge over the same dataset several times and record the variance. That number is the smallest change you can legitimately claim to detect. Reporting a movement smaller than it is reporting noise with a decimal point.

It needs to be evaluated. Label a few hundred examples by hand, run the judge over them, and measure agreement. If the judge disagrees with your humans, the judge is measuring something other than what you intended, and every downstream number inherits that error.

That last step is the one that gets skipped, and skipping it means the entire evaluation apparatus rests on an unvalidated assumption. It is also a bounded piece of work: a few hundred examples, once, plus a re-check when you change judge models.

Datasets

An evaluation dataset that only contains representative queries measures the average case, which is the case you already know works.

Four categories worth constructing deliberately:

Representative traffic, sampled from real usage. This is your baseline.

Known failures. Every production incident becomes a permanent test case. This is the highest-value category and it accumulates for free if you build the habit.

Adversarial inputs: prompt injection attempts, out-of-scope requests, ambiguous phrasings. What you are measuring here is refusal behaviour, and refusal behaviour is where deployed agents most often embarrass their owners.

Boundary cases where the correct answer is "I don't know". An agent that never says it is worse calibrated, not better, and no representative dataset will reveal that.

Version the dataset alongside the code, in the repository, using the same release discipline from Week 3. An evaluation result is only comparable to another result computed on the same dataset, so the dataset version belongs in the release evidence.

Where this connects

The reliability contract from Week 1 asked what happens when data arrives twice. The agent equivalent is what happens when the same query arrives twice and produces different answers, which users experience as a bug even though a non-deterministic system is behaving as specified. The contract question is what variation you consider acceptable and how you would detect exceeding it.

The software-product discipline from Week 3 applies unchanged. Evaluation datasets, scorer definitions, judge prompts, and pinned judge model versions are all code. They belong in the repository, under review, promoted through environments.

The integration patterns from Week 4 apply because agent invocations are long-running remote operations with exactly the properties that made the operation-resource pattern necessary.

Next week takes the other half of this piece: what a single trace can establish about an agent failure, and the three kinds of question it cannot answer no matter how well you instrument. The short version is that traces localise and evaluation judges, and confusing the two is how a team ends up reading generated reasoning text as if it were a log.

Where to start

Instrument tracing before building any evaluation. You cannot evaluate what you did not record, and traces are only collected going forward.

Write three code-based scorers for things you can check deterministically. This will be more than you expect.

Add RetrievalGroundedness and Safety to production monitoring. Neither needs labels.

Build a dataset from your known failures, which you have already, in incident tickets.

Measure your judge's noise floor before reporting any improvement.

Then, and only then, the custom judges.

Verified against Databricks documentation on 2026-07-29: Scorers and LLM judges, Built-in LLM judges, Code-based scorer examples, Tracing FAQ. Judge availability and trace limits change; check the current reference before building on any specific one.