Production Databricks Notes
Architecture decisions, investigations, and implementation patterns for data and AI systems that must operate beyond the demo.
- The Reliability Contract of a Databricks Pipeline — A pipeline becomes production when you can answer what it does when the same row arrives twice. Twelve questions, and most of them are business decisions.
- Designing an Idempotent CDC Pipeline with Lakeflow — CDC pipelines rarely throw. They produce a target table that is quietly wrong. What AUTO CDC handles for you, and the four decisions it still leaves open.
- The Sequence Column Is a Business Decision — SEQUENCE BY decides which version of a record wins, and that is a question about the business. What each of the usual timestamp candidates actually asserts.
- Treating Databricks Projects as Software Products — Ask a data team what they deploy and the answer is a notebook. What changes when a Databricks project gets a versioned artefact, a test gate, and its own identity.
- Seven Signals That a Databricks Repository Is Still a Prototype — Seven cheap checks that tell you whether a repository was promoted to production by decision or by usage, and which failure will arrive first.
- Calling Databricks from ASP.NET Core Without Building a Fragile Integration — A cold warehouse, a double-clicked button, and a deploy mid-request. The state machine and idempotency work that keeps an ASP.NET Core integration from going fragile.
- Evaluating AI Agents on Databricks — When the correct output is a set rather than a value, you cannot assert it. Code scorers, LLM judges, and where each one belongs in an evaluation.
- What an Agent Trace Reveals, and What It Cannot Prove — Tracing is the first thing teams instrument and the last thing they reason about. What a trace genuinely establishes, and the questions it cannot answer.
- Before You Trust an LLM Judge, Measure Its Noise Floor — An evaluation score moved from 0.81 to 0.84 and the change shipped. Measuring what your judge produces on unchanged input, before you trust the difference.
- Unity Catalog as a Platform Contract — Unity Catalog is introduced as access control, which leads teams to treat it as configuration. It is where every other decision in this series becomes enforceable.
- Four Ways to Isolate Tenants in Unity Catalog — Where the tenant boundary sits is expensive to reverse. Four options in Unity Catalog, strongest isolation first, with the blast radius and cost of each.
- A Reference Architecture for Production Databricks Systems — Seven checklists assembled into one system. Fourteen decisions, the couplings between them, and the order to make them in.