Production Databricks Notes: architecture decisions, investigations, implementation patterns, and lessons for data and AI systems that must operate beyond the demo.

A pipeline that transforms data correctly is not yet a production pipeline. It becomes one when you can answer what it does when the same row arrives twice, when a row arrives six hours late, when the schema gains a column overnight, when a job dies halfway through a write, and when someone asks you to reprocess last Tuesday.

Most of those answers are decisions, not features. The platform will execute whatever you specify. It cannot tell you whether a duplicate customer address is a harmless rewrite or a compliance incident. That judgment belongs to you, and writing it down is what I call the reliability contract.

I use the term deliberately. A contract has parties, obligations, and consequences for breach. The parties here are the upstream system, the pipeline, and every consumer downstream. Most pipeline incidents I have investigated came from an obligation nobody had written down, which meant nobody had noticed it was unmet.

What a reliability contract covers

Twelve questions. If your team can answer all twelve about a pipeline, that pipeline is production grade regardless of the tooling underneath. If it can answer four, you have a working transform and an unbounded liability.

flowchart LR
    S[Source] --> I[Ingestion]
    I --> V[Validation]
    V --> T[Transformation]
    T --> O[Output Tables]
    R[Retry Policy] -. controls .-> I
    C[Checkpoint State] -. controls .-> I
    Q[Quality Contract] -. controls .-> V
    REC[Recovery Procedure] -. controls .-> T
    OBS[Observability] -. monitors .-> I
    OBS -. monitors .-> V
    OBS -. monitors .-> T

Alt text: A left-to-right data flow from Source through Ingestion, Validation, and Transformation to Output Tables. Four control planes attach with dashed arrows: retry policy and checkpoint state govern ingestion, a quality contract governs validation, and a recovery procedure governs transformation. Observability monitors all three stages.

The diagram makes a point I want to state directly. The horizontal path is the part teams design carefully. The dashed vertical attachments are the part that determines whether the pipeline survives a bad week, and they are usually inherited from a template rather than chosen.

1. Input assumptions

Write down what you believe about the source before you write a line of transformation logic. At minimum: delivery guarantee, ordering guarantee, uniqueness, mutability, volume envelope, lateness bound, and what a null in an update field means.

That last one is not padding. A null arriving in an update is either the source saying the field is now empty, or the source sending a partial update where null means unchanged. Both are common, both are correct behaviours for a source system to have, and the data cannot tell you which one you are looking at. Getting it wrong in one direction silently overwrites good values; getting it wrong in the other makes it impossible to ever clear a field.

The useful discipline is to state each one as a claim that could be false, then decide what happens when it is. "Events arrive at most once" is a claim. If it turns out to be wrong at 03:00 on a Sunday, does your pipeline produce a duplicate row, a failed batch, or a quarantined record? You will answer that question eventually. Answering it in a design document costs an hour. Answering it during an incident costs considerably more.

I have never seen a source system that met all of its stated guarantees under load. That is not a criticism of source systems. It is a reason to design for the violation.

2. Idempotency

Reprocessing the same input must converge to the same output. This sounds obvious and is routinely violated by pipelines that append.

The three common shapes:

Append-only with a natural key and a dedup step. Cheap to reason about, and the deduplication window becomes a parameter you must justify. A seven day window means a duplicate arriving on day eight lands silently.

Merge on a business key. Convergent by construction, at the cost of a read-modify-write against the target. The subtlety is what happens when two versions of the same key arrive in the same batch, which is what the sequence column is for.

Overwrite a partition. Simple and correct when the partition is the unit of reprocessing. It fails quietly when late data belongs to a partition you already rewrote.

For the second and third shapes, the question I ask is which field decides the winner when two records collide, and whether that field comes from the source system or from ingestion time. Those are different fields with different failure modes, and the choice is business semantics rather than configuration. I will treat that specific problem separately in a follow-up note, because it deserves more room than a section.

3. Checkpoint and state ownership

A streaming pipeline keeps a checkpoint that records what it has consumed and, for stateful operations, what it is holding in memory between micro-batches. Two things follow.

First, the checkpoint is production state. It is not a cache and not a temporary file. Deleting it does not reset a pipeline to a clean state, it resets the pipeline's belief about what it has already processed, which is a different and more dangerous thing.

Second, somebody owns it. Name them. The failure I have seen most often is a developer clearing a checkpoint to unblock a stuck pipeline in an environment where that action reprocesses six weeks of data into a table that other teams query.

Your contract should state where the checkpoint lives, who may delete it, what happens to downstream tables if it is deleted, and whether the pipeline is safe to run twice concurrently. If you do not know the answer to the last one, assume it is not.

4. Duplicate processing

Distinguish two cases that get conflated.

A duplicate event means the source emitted the same logical change more than once. Deduplication by event identifier handles it, and the identifier has to come from the source rather than being computed downstream from fields that might legitimately repeat.

A replayed batch means your pipeline processed the same input twice, usually after a retry. Idempotent writes handle it. Deduplication by event identifier does not, because the second pass sees the same identifiers and the dedup state may already have advanced past them.

A pipeline can be safe against one and exposed to the other. Testing only the first is the more common gap, because duplicate events are easy to imagine and retried batches only appear when something else has already gone wrong.

5. Data-quality behaviour

Every quality rule needs an action attached, chosen from three: drop the record, fail the batch, or route it to quarantine and continue.

The choice is a business decision wearing technical clothing. Dropping is right for records that are noise. Failing is right when partial data is worse than no data, which is true more often than teams expect for financial and regulatory outputs. Quarantining is right when you need the record back later, and it carries an obligation most teams skip: somebody has to look at the quarantine. A quarantine table nobody reads is a deletion with extra steps and a larger storage bill.

State the expected quarantine rate. An unexpected zero is as much a signal as a spike, and usually means the rule stopped matching rather than the data getting cleaner.

Lakeflow expectations give you two of the three actions directly:

from pyspark import pipelines as dp

@dp.table()
@dp.expect("age_valid", "age > 0")                          # keep the row, log the violation
@dp.expect_or_drop("email_required", "email IS NOT NULL")   # drop the row
@dp.expect_or_fail("price_positive", "price >= 0")          # fail the pipeline update
def my_dataset():
    return spark.sql("SELECT * FROM source_table")

expect_all, expect_all_or_drop, and expect_all_or_fail take a dict and apply one action to a group of rules.

Notice what is missing. There is no expect_or_quarantine. Drop and fail are one decorator each; quarantine is a pattern you build, usually by expecting the inverse condition into a separate table and reading it on a schedule.

That asymmetry is worth knowing when you write the contract, because the cheapest option to implement is expect_or_drop and the one that preserves the most information is the one nobody ships. If your contract says quarantine, budget for building it, and budget for the person who reads it.

6. Retry versus replay

These get used interchangeably and should not be.

Retry repeats a failed unit of work in place. It assumes the failure was transient and that nothing partial escaped. It is safe when writes are atomic or idempotent, and unsafe when a job writes to two tables without a transaction spanning both.

Replay deliberately reprocesses input that already succeeded, to correct a logic error or fill a gap. It assumes the outputs are reconstructible from the inputs, which is only true if you kept the inputs and if the transformation is deterministic. A transformation that calls a model endpoint or reads a mutable lookup table is not deterministic across time, and its replay produces a different answer that is not obviously wrong.

Your contract should say which operations are safe to retry automatically, which require a human, and what the replay procedure is including its blast radius. "We would just rerun it" is not a procedure.

7. Partial failure

A job that writes three tables and dies after the second leaves the system in a state your downstream consumers were not told about. Two defensible answers exist.

Make the write atomic across the affected tables, which usually means restructuring so there is one commit point. Or make the intermediate state legible, so consumers can tell the difference between "table B is empty" and "table B has not been written yet for this run". Publishing a run identifier and a completion marker alongside the data costs little and removes an entire class of ambiguity.

The answer I would push back on is treating partial failure as rare enough to handle manually. It scales with the number of pipelines, and manual handling scales with the number of engineers.

8. Schema evolution

Decide in advance what happens for each of: a new nullable column, a new required column, a widened type, a narrowed type, a renamed column, and a dropped column.

Additive changes can usually flow through. The rest deserve a failure by default, because a pipeline that silently accommodates a narrowed type is a pipeline that silently truncates. My default is to fail closed on anything that is not purely additive, then relax it where a specific case justifies the risk.

Whatever you choose, the important property is that a schema change produces a signal. Silent accommodation is the failure mode, because the damage is discovered by a consumer weeks later and traced back through a month of data.

9. Recovery point and recovery time

Two numbers, stated explicitly, agreed with whoever depends on the output.

Recovery point is how much data you can afford to lose. Recovery time is how long the output may be stale or unavailable. Both are business inputs. Neither is a property of the platform.

Then verify them. An untested recovery target is an aspiration. Restore the pipeline into a scratch environment from a real backup and time it. The number you get is usually larger than the number in the document, and the gap is worth knowing before an incident rather than during one.

10. Observability

Three layers, and most teams build only the first.

Execution: did the job run, how long did it take, did it fail. Every platform gives you this.

Data: how many rows arrived, how many were dropped or quarantined, how far behind is the stream, when was the output last updated. This is where problems become visible before consumers notice.

Business: does the output still make sense. Row counts within an expected band, a total that reconciles against a source of truth, a distribution that has not shifted. A pipeline can be green at the execution layer and producing nonsense, and only the business layer catches it.

The measure worth adding to your contract is freshness, expressed the way consumers experience it. "Orders are visible in the serving table within fifteen minutes of being placed" is a statement a business owner can agree or object to. "The job runs every ten minutes" is not.

11. Ownership and escalation

Name the owning team, the escalation path outside working hours, and the maximum time the pipeline may stay broken before someone is woken up. Then name who is allowed to decide that the output is wrong and should be withdrawn from consumers.

That last one gets forgotten and matters most. During an incident the expensive question is rarely how to fix the pipeline. It is whether the data already published should be retracted, and who has the authority to say so.

12. Change safety

The contract needs a clause about itself. Which changes to this pipeline require a full replay, which require consumer notification, and which can ship silently. A transformation change that alters historical output is a different kind of release from a performance tweak, and treating them identically means treating both as the risky one or both as the safe one.

The checklist

Take this to a design review. Twelve questions, one per contract section, answerable in a sentence each. Any question that produces a discussion instead of an answer is where the risk lives.

What do you believe about the source, and what happens when each belief turns out to be false? Include what a null in an update field means: empty, or unchanged.
What does the pipeline do when the same record arrives twice?
Who owns the checkpoint, and what breaks downstream if it is deleted?
Which field decides the winner when two versions of a key collide, and does it come from the source or from ingestion?
For each quality rule: drop, fail, or quarantine, and who reads the quarantine?
Which failures retry automatically, which require a human, and what is the replay procedure and its blast radius?
What state is the system in if the job dies between two writes?
What happens on a new required column, and on a narrowed type?
What are the recovery point and recovery time targets, and when were they last tested?
What is the freshness guarantee, stated in terms a consumer would recognise?
Who is paged when this breaks at 03:00, and who is allowed to decide that published output was wrong and must be withdrawn?
Which changes require a replay, and which require telling consumers before you ship them?

Question 11 is the one that gets skipped and the one you will want first. During an incident the expensive question is rarely technical.

Where this goes next

The next piece implements the first four answers with Lakeflow's CDC APIs, against a synthetic change stream containing duplicates, reordering, late updates, and a replay after interruption, with tests that assert the final state after each. Writing a contract is the easy half. Demonstrating that an implementation satisfies it under adversarial input is the half that produces evidence.

The same framing extends past data pipelines. An API that wraps a long-running platform operation has a reliability contract. So does a retrieval-augmented agent, which is where Week 5 picks this up: the question shifts from "is the row correct" to "which failures are unacceptable", "correct" becomes a range rather than a value, and the thing checking it is another model.

Between them sits the question of where a contract lives once it is written, which is Week 3's subject and the reason a document nobody can find is worth about as much as a decision nobody made.

This is an engineering recommendation rather than a Databricks platform guarantee. Verified against Databricks documentation on 2026-07-29; platform behaviour changes, and I will update this article rather than let it go stale. If you disagree with the defaults I have argued for, I would rather hear it in the comments than not.