An investigation into the difference between observing an agent and understanding it

Production Databricks Notes: architecture decisions, investigations, implementation patterns, and lessons for data and AI systems that must operate beyond the demo.

Tracing is the first thing everyone instruments and the last thing anyone reasons carefully about.

The pitch is straightforward: record the agent's execution, and when something goes wrong you can see what happened. This is true and it is the reason to instrument tracing before anything else.

A stronger claim gets attached to it: that because you can see the execution, you can explain the outcome. That one does not hold, and the gap between the two is where a good deal of wasted debugging time goes.

This piece separates what a trace observes from what it lets you infer.

Week 5 argued that traces should be the unit of evaluation, because scoring a final response cannot distinguish an agent that retrieved the wrong documents from one that reasoned badly over the right ones. That argument stands. This piece is the limit case of it: traces are the right unit to evaluate, and there is a specific and useful boundary on what any single one of them can prove.

What a trace is

A trace is a structured record of one execution: the input, each retrieval, each tool call with its arguments and results, intermediate model outputs, and the final response, arranged as spans with timing and parentage.

Four operational facts from the Databricks documentation that shape how you can use it:

Experiments not in Unity Catalog are capped at 100,000 traces total. The documented remedy is to put the experiment in Unity Catalog, which the docs describe as giving long-term retention of large trace volumes with no per-experiment limit.

Production monitoring is in Beta at the time of writing, per the banner on its documentation page, and it is what the docs point you at for large-scale trace analysis.

The trace list returns the most recent 1,000 traces by default. An older trace outside that window will not appear in search results even when you match its trace ID exactly.

Ingestion is rate limited, initially to 200 traces per second per workspace and 100 MB per second per table. A chatty agent under load can reach the first of those, and the failure looks like missing traces rather than an error in your application.

The 1,000-trace default is the one that produces the most confusion in practice. Someone has a trace ID from an incident, searches for it, gets nothing, and concludes the trace was never written. It was; it is outside the default window. Query the Delta table directly rather than the trace list for anything historical, which is also the answer to "how do I analyse traces in aggregate".

flowchart TB
    T["One agent trace"]

    T --> A["TIER 1: observed directly<br/>tool calls, with arguments and order<br/>tool results, including handled errors<br/>retrieved documents and scores<br/>per-span timing<br/>final output"]

    T --> B["TIER 2: strong inference<br/>retrieval failure vs generation failure<br/>redundant or repeated tool calls<br/>whether the context held the answer<br/>structural vs incidental latency"]

    T --> C["TIER 3: not establishable<br/>why the model produced this output<br/>whether the output is correct<br/>whether the failure is systematic<br/>what upstream filtering removed"]

    C -.needs.-> X["experiments, evaluation,<br/>aggregate queries,<br/>the upstream data contract"]

Alt text: One agent trace splits into three tiers. Tier 1, observed directly: tool calls with arguments and order, tool results including handled errors, retrieved documents and scores, per-span timing, final output. Tier 2, strong inference: retrieval failure versus generation failure, redundant tool calls, whether the context held the answer, structural versus incidental latency. Tier 3, not establishable: why the model produced this output, whether the output is correct, whether the failure is systematic, what upstream filtering removed. Tier 3 needs experiments, evaluation, aggregate queries, and the upstream data contract.

Tier one: what a trace directly observes

These are facts. The trace recorded them and they are not in dispute.

Which tools were called, with which arguments, in which order. This is the highest-value part of a trace and it is frequently under-used. Most agent failures I have investigated are visible here before they are visible anywhere else.

What each tool returned. Including the error, which is the case people forget to look at because the agent recovered and produced an answer anyway.

What was retrieved. Document identifiers, scores, and the text that entered the context.

Timing per span. Where the latency went.

The final output. What the user saw.

If your question can be answered from this list, tracing answers it completely. "Did it call the pricing tool?" "How many documents did it retrieve?" "Which step took nine seconds?" These are observation questions and they are settled by looking.

Tier two: what a trace supports as strong inference

Not directly recorded, but well supported by what is.

Whether the failure was in retrieval or in generation. If the retrieved documents do not contain the answer, generation was working with what it had. This inference is strong, and it is the single most useful thing tracing buys you, because it splits your debugging in half immediately.

Whether a tool was called redundantly. Repeated identical calls in one trace are visible and usually mean the agent did not register the first result.

Whether the agent had the information it needed. You can read the context and check. This is almost a tier-one observation, with one caveat covered below.

Whether latency is structural or incidental. Consistent slowness in one span across many traces is structural. One slow span in one trace is not evidence of anything.

Inferences at this tier are reliable enough to act on. They are still inferences, which matters when someone asks how you know.

Tier three: what a trace cannot establish

This is the part worth being precise about.

Why the model produced this output. The trace records that the model emitted certain tokens given certain context. It does not record why. Intermediate reasoning text, when present, is generated output, not a log of the computation that produced the answer. It is evidence about what the model produced, not a causal account. Treating it as one is the most common overreach in agent debugging, and it is seductive because the text reads exactly like an explanation.

Whether the output is correct. A trace shows what was produced. Correctness requires a comparison against something outside the trace: ground truth, a judge, a human. This is why evaluation is a separate apparatus rather than a view over your traces.

What would have happened otherwise. A trace is one execution. It says nothing about the counterfactual. Whether a different retrieval would have produced a better answer is an experiment, not a trace query.

Whether the failure is systematic. One trace is one sample. This is the inference error I see most often in incident reviews: a trace is found, a root cause is identified, a fix is shipped, and nobody checked how many other traces show the same pattern. Sometimes the answer is one.

Whether the agent will do this again. Non-determinism means a trace records what happened once. Reproduction requires re-running, and re-running may not reproduce.

What the user actually wanted. The trace records the input. Intent is not in it.

The one that reaches furthest

A trace cannot show you what was excluded before the trace began.

If retrieval returned five documents, the trace shows five documents. It does not show the document that was not indexed, the record the quality contract dropped at ingestion, or the row a tenant filter removed. Those are absences, and absences do not appear in a record of what happened.

This connects directly to Week 1. A pipeline that quarantines malformed records is behaving correctly and is also silently narrowing what any downstream agent can know. The agent then answers from a filtered view of reality, the trace shows a clean execution over the context it was given, and RetrievalGroundedness scores it well, because the response is faithful to that context.

Every component is working. The system is wrong. Nothing in the trace indicates it.

The only defence is knowing your upstream contract. This is why the reliability contract from Week 1 turns out to matter for an AI system built five layers above it, and it is why "what did the quality rule drop" belongs on your incident checklist for agent failures even though it looks like a data engineering question.

A trace review procedure

When investigating an agent failure, in this order:

Read the input first, before the output. Reading the output first anchors you on the failure and you will read everything afterwards looking for a cause for that specific failure.

Check what was retrieved and whether the answer is in it. This splits the problem in half and takes thirty seconds.

Read the tool calls in order, including arguments. Wrong arguments to a correct tool is a common failure and it looks like a reasoning failure until you look at the arguments.

Check tool errors, especially ones the agent recovered from. A recovered error that produced a degraded answer looks like success in every aggregate metric.

Now read the output.

Then find how many other traces show the same shape. This is the step that most often changes the conclusion, and it is the step most often skipped because by this point you feel finished.

That last step is what Delta-backed trace storage exists for. Ad-hoc trace inspection is a UI activity; pattern confirmation is a query, and it looks like ordinary analytics because it is:

-- How common is the pattern you just found in one trace?
SELECT
  date_trunc('day', request_time)          AS day,
  count(*)                                 AS traces,
  count_if(retrieved_count = 0)            AS empty_retrievals,
  count_if(retrieved_count = 0) / count(*) AS empty_rate
FROM   my_agent_traces
WHERE  request_time >= current_date() - INTERVAL 14 DAYS
GROUP  BY 1
ORDER  BY 1;

Two notes before you copy that. The exact table name and column layout depend on your experiment configuration, so read yours rather than assuming mine; the shape of the question is what transfers, not the schema. And a rate is the right output, not a count, because traffic volume moves and a rising count against rising traffic is not a regression.

This query is the difference between "I found a trace where retrieval returned nothing" and "retrieval returns nothing on 4% of traffic, up from 0.2% last Tuesday". Only the second one is worth an incident.

What to instrument

For a supported library, autologging gets you most of this without writing anything:

import mlflow

mlflow.langchain.autolog()      # 40+ libraries have an equivalent

Autolog gives you the framework's own boundaries. It does not know which of your functions are meaningful, so anything you want attributed separately needs a span of its own. The decorator is the cheap version:

import mlflow
from mlflow.entities import SpanType

@mlflow.trace(
    name="retrieve-policy-docs",
    span_type=SpanType.RETRIEVER,
    attributes={"index": "policies-v4", "top_k": 8},
)
def retrieve(query: str) -> list[Document]:
    ...

And the context manager when you need to set inputs and outputs explicitly, or when the unit you care about is not a function:

with mlflow.start_span(name="rerank") as span:
    span.set_inputs({"query": query, "candidates": len(candidates)})
    ranked = reranker(query, candidates)
    span.set_outputs({"kept": [d.id for d in ranked]})

Note what went into attributes and set_outputs above: the index version, and document identifiers rather than document text. Both are chosen so that a future aggregate query can group by them. A span that records only prose is readable and not queryable.

Trace boundaries you will want, from experience of not having them:

Retrieval as its own span, with document identifiers and scores, not just the concatenated text. The identifiers are what let you correlate across traces.

Each tool call as its own span, with full arguments. Truncated arguments are the most common regret.

Any point where the agent decides between paths. If a routing decision is inside an opaque span, every failure downstream of it is harder to attribute.

Whatever version identifiers you have: prompt version, model version, retrieval index version. When a regression appears, the first question is what changed, and the trace is where the answer should already be.

Traces are only collected going forward. Everything you did not instrument before an incident is information you do not have during it, which is the argument for instrumenting more than feels necessary now.

The honest summary

Tracing tells you what happened. It supports strong inference about where something went wrong. It cannot tell you why the model behaved as it did, whether the output was correct, or whether the problem is systematic.

Those three require, respectively: experiments, evaluation, and aggregate queries.

A team that understands this uses traces to localise a problem and then reaches for the right tool. A team that does not spends its time reading reasoning text looking for an explanation that is not in there.

Verified against Databricks documentation on 2026-07-29: Tracing FAQ, Trace agents deployed on Databricks. Limits and quotas change and higher production limits are a conversation with Databricks support rather than a setting. The three-tier framing above is my own and is engineering judgement rather than documented guidance.