The decisions from seven weeks, assembled into one system and ordered by what to fix first

Production Databricks Notes: architecture decisions, investigations, implementation patterns, and lessons for data and AI systems that must operate beyond the demo.

The previous seven pieces each addressed one layer. Reliability contracts, CDC design, software delivery discipline, application integration, agent evaluation, what traces can and cannot prove, Unity Catalog as a platform contract.

Read separately they are seven checklists. Assembled they are one system, and the assembly reveals something the individual pieces do not: the decisions are not independent. Several of them constrain each other in ways that are invisible until you make one and discover it has removed an option somewhere else.

This piece puts them together, names the couplings, and gives an order.

The system

flowchart TB
    CDC[CDC Feeds] --> BRONZE
    API[External APIs] --> BRONZE
    FILE[File Drops] --> BRONZE

    BRONZE[Bronze / Raw] --> CONTRACT
    CONTRACT["Quality Contract<br/>drop / fail / quarantine"] --> SILVER
    SILVER["Silver / Conformed<br/>AUTO CDC flows"] --> GOLD
    GOLD[Gold / Serving] --> APP[ASP.NET Application]
    GOLD --> AGENT[AI Agent]
    GOLD --> BI[BI and Reporting]

    UC["Unity Catalog<br/>tenant boundary, lineage, audit"] -.governs.-> SILVER
    BUNDLE["Declarative Automation Bundle<br/>code, resources, permissions"] -.deploys.-> CONTRACT
    EVAL["Evaluation and Monitoring"] -.monitors.-> AGENT

Alt text: A top-to-bottom architecture diagram. CDC feeds, external APIs, and file drops flow into Bronze / Raw, then through a Quality Contract (drop, fail, or quarantine) into Silver / Conformed where AUTO CDC flows run, then into Gold / Serving, which feeds an ASP.NET application, an AI agent, and BI and reporting. Three dashed attachments sit alongside the spine: a Declarative Automation Bundle deploys the quality contract, Unity Catalog governs Silver, and evaluation and monitoring watches the AI agent.

Nothing here is novel. The medallion pattern is well established. What the diagram is for is locating the decisions.

The decision register

Every one of these has to be answered by somebody. The failure is not answering them badly; it is not noticing they were questions.

#	Decision	Where it was covered	Cost of deferring
1	What happens when a record arrives twice	Week 1	Silent duplicates, discovered by a business user
2	Which field decides version order	Weeks 1, 2	Non-deterministic output on replay
3	Drop, fail, or quarantine per quality rule	Week 1	Quarantine tables nobody reads
4	Retry versus replay, and replay's blast radius	Weeks 1, 4	Recovery decided under time pressure
5	Type 1 or Type 2, and what triggers a version	Week 2	History that does not exist when asked for
6	What a delete means	Week 2	Delete events landing as ordinary rows
7	Null in an update: empty or unchanged	Week 2	Good values silently overwritten
8	Which identity runs what	Weeks 3, 7	Outage caused by an HR event
9	What the deployment unit is	Week 3	Nothing describes production
10	How rollback works for data already written	Weeks 3, 7	No recovery path
11	Sync or operation-resource for external calls	Week 4	Timeouts, duplicate submissions
12	Where the tenant boundary sits	Week 7	Expensive to move later
13	What you evaluate, and against what dataset	Weeks 5, 6	Improvements that are noise
14	Which judge model, pinned to which version	Week 6	Historical comparisons silently invalid

Fourteen decisions. Most projects have explicit answers to four or five and implicit answers to the rest, where "implicit" means whatever the default was.

Scoring your own system

Score each decision 0, 1, or 2. The scale is deliberately about where the answer lives, not whether the answer is good, because a written wrong answer gets corrected in review and an unwritten right answer evaporates when its author changes teams.

0: nobody has decided. The behaviour is whatever the platform default produces. Nobody can tell you what it is without reading code.

1: somebody knows. A person can answer correctly from memory. It is not written anywhere a new joiner or an on-call engineer would find it.

2: it is written down, owned, and tested. There is a document naming the answer and the person accountable for it, and something in CI fails if the system stops matching it.

The gap between 1 and 2 is where most teams sit and it is the gap that matters. A decision at 1 is correct until the person holding it is unavailable, which is precisely the moment you need it.

Rough reading of the total, out of 28:

Under 10. The system is running on defaults. This is normal for something that grew out of a prototype and it is not an emergency by itself, but you cannot make a reliability claim about it and should not let anyone else make one on your behalf.

10 to 20. In transition. The useful move is to find which unanswered decisions gate others, per the ordering below, rather than working left to right through the table.

Above 20. The remaining risk is probably somewhere this series does not cover: cost, latency, disaster recovery, or organisational rather than technical.

Two scoring notes. Decision 8, identity, should be weighted double, because it gates three others. And any decision scoring 2 whose test has not run in the last quarter is scoring 1, because an untested assertion about failure behaviour is a belief, and the whole argument of this series is that beliefs about failure behaviour are usually wrong.

The couplings

This is the part that only appears when the layers sit next to each other.

Sequence column → history table type. Decision 2 determines decision 5's schema. SCD Type 2 requires __START_AT and __END_AT to have the same data type as the sequence column. Choosing a BIGINT log sequence number gives you a correct history table whose validity intervals are unreadable to anyone outside the data team.

Tenant isolation → rollback mechanism. Decision 12 can remove decision 10's implementation. Time travel does not work on tables with row-level security or column masks. A table that needs both tenant isolation and time-travel recovery needs its isolation to be structural rather than a predicate.

VACUUM threshold → recovery objective. Decision 10 has a second constraint that is easy to miss because two settings disagree. delta.logRetentionDuration defaults to 30 days, delta.deletedFileRetentionDuration defaults to 7, and the shorter one is your actual rollback window. A restore to a point 14 days ago fails against a log entry that still exists, because the files it references are gone. If your stated recovery objective exceeds seven days and nobody has raised file retention, the objective is a number in a document rather than a property of the system.

Tenant isolation → RAG feasibility. Decision 12 also reaches decision 13. You cannot create an AI Search index from a table with row filters or column masks applied. If tenant-scoped data feeds an agent, the boundary has to sit at the index level or above.

Identity → everything in governance. Decision 8 gates decisions 12 and 13. Row filters and grants key on the executing principal. A pipeline running as its author has the author's permissions, which are broader than the workload's, so the isolation you designed is not the isolation you have. It is the decision this series kept colliding with: the Week 3, Week 4, and Week 7 articles each hit it, and both of their companion notes do too.

Deployment unit → evaluation validity. Decision 9 gates decision 13. An evaluation result is only comparable to another computed on the same dataset with the same judge model. If those are not versioned with the code, your evaluation history is a series of numbers without units.

Quality-rule action → replay semantics. Decision 3 quietly breaks decision 4. Replay assumes you can reconstruct the output from the input, but a quarantined record is no longer in the input you replay from, and a dropped one never was. So a replay after a quality incident reproduces the same gap, faithfully, and the pipeline reports success. If your replay procedure is meant to be a recovery mechanism, it has to read from a layer upstream of where the quality rules ran, which usually means bronze rather than silver, and which is a decision to make when you design the medallion split rather than during the incident.

mode: production → pipeline runtime behaviour. Decision 9 reaches further than deployment. Deploying a bundle with mode: production validates that every pipeline is marked development: false, so your deployment target and your pipeline's runtime configuration are coupled by the tooling. The failure is benign and confusing: a bundle that deploys cleanly from a laptop fails in CI, because mode: development set the flag that mode: production rejects.

Quality contract → agent groundedness. Decision 3 also propagates all the way to decision 13. An agent grounded in a table that silently drops malformed records is grounded in a filtered view of reality, and RetrievalGroundedness will score it well, because the response is faithful to the context it was given. The judge cannot see what the quality rule removed.

That last one is the coupling I would flag hardest, because it crosses the furthest and every component involved is behaving correctly.

An order

Not a maturity model. An order of operations, sequenced by what unblocks the most other work.

First: identity. Decision 8. Move production execution to service principals, separate the deployment identity from the workload identity. It is usually the cheapest change on this list and it gates the governance and isolation work.

Second: the deployment unit. Decision 9. Extract logic from notebooks into importable modules, add a bundle definition covering what exists, run databricks bundle validate in CI. You are not changing behaviour, you are making the current state describable, which is a precondition for changing it deliberately.

Third: the tenant boundary. Decision 12. Do this before the data volume makes it expensive and before you have picked an isolation mechanism that removes an option you will want.

Fourth: the reliability contract. Decisions 1 through 7. Written down. Most are business questions and the delay is getting answers, not implementing them, so start asking early.

Fifth: failure tests. The test that processes the same events twice. The test that shuffles order. The test that kills the worker mid-submission. These catch the failures that the happy path structurally cannot.

Sixth: integration patterns. Decision 11, when an external application enters the picture.

Seventh: evaluation. Decisions 13 and 14, when agents do. Instrument tracing first, because traces only accumulate going forward.

The order matters more than the pace. Doing the reliability contract before the identity work means writing down guarantees the permission model cannot enforce.

What this does not cover

Cost management. Important and mostly orthogonal to correctness. The one place they touch is the VACUUM setting above, where a longer recovery window costs storage, and that trade is worth making explicitly rather than by inheriting a default.

Streaming latency optimisation. Everything here assumes minutes are acceptable. If you need sub-second, the architecture changes shape and several recommendations stop applying, starting with the scheduled, sampled nature of trace scoring, which makes agent monitoring near-real-time rather than real-time.

Disaster recovery across regions. A metastore is regional. That single fact drives most of the design, and doing it justice means a piece about replication, failover testing, and what "recovered" means for a system whose state includes checkpoints and a transaction log.

Fine-tuning and model customisation. A different discipline with its own evaluation problems, though the judge-validation argument from Week 6 transfers unchanged.

Real-time model serving. Adjacent to Week 4's integration patterns and different enough in its latency and scaling behaviour to deserve separate treatment.

Each would need its own investigation to write about honestly rather than plausibly, which is the line this series has tried to hold. What it covers instead is the layer underneath all five: the decisions that determine whether the system is correct before it is fast, cheap, or resilient.

Two things this deliberately does not prescribe, either, because they are context-dependent and the confident answers you will read elsewhere are usually generalised from one company's constraints: whether to use notebooks in production at all (yes, if they are thin entry points over tested modules), and how many environments to run (as many as you have distinct approval gates, which for most teams is three and for some is two).

The one-page version

If you take one thing from eight weeks:

Write down the decisions. Not the architecture, the decisions. Fourteen questions with fourteen named owners and fourteen written answers.

Systems do not fail because a team chose Type 1 over Type 2. They fail because nobody chose, the default applied, and eighteen months later someone asks for a history that does not exist.

The platform will execute whatever you specify. It cannot tell you what to specify, and it will not tell you that you never did.

What ships alongside this

A template repository, published at github.com/ivanvyd/production-databricks-patterns, with the structure from Week 3: src/ holding importable Python that tests without a cluster, a bundle definition with dev, staging, and production targets, the identity separation, and the failure tests from Weeks 2 and 4: the one that processes the same events twice, the one that shuffles their order, and the one that kills the worker mid-submission. It is a starting point rather than a framework, and it is deliberately small enough to read in one sitting.

A written walkthrough of the promotion path, in the same repository at docs/walkthrough.md: one change moving from a laptop through validation, tests, plan review, staging, and into production, with the commands at each step and the release evidence it leaves behind.

If you score your own system against the register above, I would like to see what you find, particularly where it disagrees with the ordering I have argued for. The couplings section is the part of this series I am least confident is complete, and the way to find the ones I have missed is for someone to hit one I did not list.

Series index, articles: Week 1 The Reliability Contract of a Databricks Pipeline · Week 2 Designing an Idempotent CDC Pipeline with Lakeflow · Week 3 Treating Databricks Projects as Software Products · Week 4 Calling Databricks from ASP.NET Core · Week 5 Evaluating AI Agents on Databricks · Week 6 What an Agent Trace Reveals, and What It Cannot Prove · Week 7 Unity Catalog as a Platform Contract.

Notes: The Sequence Column Is a Business Decision · Seven Signals That a Databricks Repository Is Still a Prototype · Before You Trust an LLM Judge, Measure Its Noise Floor · Four Ways to Isolate Tenants in Unity Catalog.

Platform behaviours referenced here were verified against Databricks documentation on 2026-07-29 and are cited in the individual articles. Several have changed before. Check the current reference before relying on any specific one.