Production Databricks Notes: architecture decisions, investigations, implementation patterns, and
lessons for data and AI systems that must operate beyond the demo.
Ask a data team what they deploy and the answer is often a notebook. Ask a backend team the same
question and you get a versioned artefact with a build number, a test suite that gated it, an
environment it was promoted through, and an identity it runs as.
The gap between those two answers is most of what separates a Databricks project that scales from
one that becomes load-bearing by accident.
The argument here is narrow: the unit of delivery should be a versioned, testable, deployable
system containing code, resource definitions, permissions, tests, and operational metadata.
Databricks now ships a first-party mechanism for exactly this, and the interesting work is in what
you put around it.
This follows directly from Week 1. That piece argued for writing down twelve reliability decisions:
what happens on a duplicate, which field decides version order, whether a quality violation drops or
fails or quarantines. Writing them down is necessary and it is not sufficient, because a contract
that lives in a document drifts away from the system it describes and nothing announces when it has.
A reliability contract needs a home where it can be reviewed, versioned, and tested against the
thing it governs. That home is the repository, which is what this piece is about.
A naming note, because it will confuse you otherwise
Databricks Asset Bundles were renamed Declarative Automation Bundles on 16 March 2026. The
change is non-breaking: the CLI command is still databricks bundle, existing configuration is
unchanged, and the DAB acronym survives. Databricks stated the rename was because "assets" carried
more than one meaning in the platform.
You will find both names in documentation, blog posts, and Stack Overflow answers for a long time.
They are the same thing.
Why notebook-only deployment stops working
Not because notebooks are bad. They are an excellent medium for exploration and a reasonable one for
some production workloads. The problem is using the notebook as the deployment unit, which
creates four specific costs.
Review becomes reading output. A notebook diff is a diff of serialised cells with execution
metadata mixed in. Reviewers stop reading it and start trusting the screenshot of the result.
Testing requires a cluster. When transformation logic lives in cells rather than in importable
modules, the only way to test it is to execute the notebook. That makes tests slow, expensive, and
too painful to run per commit, so they run rarely.
Deployment is imperative. Jobs, permissions, and cluster configuration get created by clicking.
Nothing in the repository states what production looks like, so nothing can detect that it drifted.
The blast radius is unbounded. Without an environment boundary that is enforced rather than
observed, a notebook pointed at the wrong catalog does production damage from a development session.
Repository boundaries
One repository per deployable system, where a system is a set of resources that release together and
share an owner. Not one per notebook, and not one for the whole data platform.
The test I use: if two things must always deploy at the same time to stay consistent, they belong in
one repository. If they can release independently and one team owns each, they should not be
coupled by a shared deployment.
production-databricks-patterns/
├── databricks.yml # bundle definition and targets
├── src/ # importable Python, unit-testable without a cluster
│ ├── transforms.py # the logic worth testing
│ └── pipelines/ # thin entry points that call into it
├── resources/ # job, pipeline, and permission definitions
├── tests/
│ ├── unit/ # no cluster required
│ └── integration/ # runs against a real workspace
├── docs/
│ ├── decisions/ # architecture decision records
│ └── evidence/ # release evidence, test output
└── .github/workflows/
Clone the layout above from
github.com/ivanvyd/production-databricks-patterns.
The unit suite runs there without a workspace or a cluster.
The property that matters: src/ contains plain Python that imports and tests without a Spark
session where possible, and notebooks become thin entry points that call into it. That single change
converts your test suite from "slow, needs infrastructure" to "runs in CI in seconds", which decides
whether tests run per commit or per quarter.
Bundle structure and targets
A bundle declares resources once and parameterises them per environment. The shape:
bundle:
name: production-databricks-patterns
variables:
catalog:
description: Unity Catalog catalog for this environment
notifications_email:
description: Where failures are sent
targets:
dev:
mode: development
default: true
variables:
catalog: dev_analytics
notifications_email: ivan@example.com
staging:
mode: production
variables:
catalog: staging_analytics
notifications_email: data-alerts@example.com
prod:
mode: production
variables:
catalog: prod_analytics
notifications_email: data-oncall@example.com
run_as:
service_principal_name: sp-data-platform-prod
mode is doing more work than it looks like. The documented differences:
|
development |
production |
| Naming |
prefixes [dev <short_name>], adds a dev tag |
no prefix |
| Schedules and triggers |
paused; override per job with schedule.pause_status: UNPAUSED |
run as configured |
| Concurrent runs |
enabled by default; override with max_concurrent_runs: 1 |
as configured |
| Lakeflow pipelines |
marked development: true |
validates all pipelines are development: false |
| Deployment lock |
disabled; re-enable with bundle.deployment.lock.enabled: true |
enabled |
| Extra prod checks |
none |
git branch must match (override with --force); run_as/permissions required unless deploying as a service principal; artifact_path, file_path, root_path, state_path must not be user-specific |
Two of these prevent incidents rather than inconvenience.
Paused schedules in development mean your personal copy of a job does not run on its production
cadence against whatever it happens to be pointed at. I have twice seen a developer copy of an
hourly job run against production for a weekend before anyone noticed.
The production check that all pipelines are development: false is a validation, not a
transformation. It fails the deploy rather than silently fixing the setting, which is the correct
choice and occasionally a surprising one.
The prod-only path constraints are the reason a bundle that deploys cleanly for you can fail in CI:
paths resolved relative to a user are exactly what mode: production rejects.
The commands
databricks bundle validate -t prod # parse, resolve variables, check against the API
databricks bundle plan -t prod # show what deploy would change
databricks bundle deploy -t prod # create or update resources
databricks bundle run -t prod job_name
databricks bundle destroy -t dev # tear down; you want this in dev, never wired to prod
plan is the one people miss. It sits alongside deploy and shows what a deployment would change
before it changes it, which is the difference between a reviewed production change and a hopeful
one. deploy also accepts --plan <path>, so a plan reviewed in CI is the plan that gets applied. Two
constraints come with it: generate the file with databricks bundle plan -o json, and the flag is
direct-engine only.
validate in CI on every pull request is the cheapest quality gate available. It catches a
misspelled resource type and an unresolved variable before anything is created. Pair it with plan
on the pull request when you want to see the resulting actions as well as the syntax, since the two
answer different questions.
Deployment identity
This is the section I would read first if I only read one.
Three distinct identities, and conflating them is the most common structural mistake:
Human identity does interactive development. It should have broad rights in development and
read-only or no rights in production.
Workload identity runs the job. It owns the execution and needs exactly the permissions the
workload requires, which is usually narrower than the developer's.
Deployment identity creates and updates resources. It needs rights to manage jobs and pipelines,
and it does not need to read the data those jobs process.
When all three are the same person, you get the failure from the prototype-signals note: the
pipeline breaks when they change teams, and the audit log cannot distinguish a human action from an
automated one at the moment you need it to.
Set run_as on production targets to a service principal. Authenticate CI with OAuth
machine-to-machine credentials rather than a personal access token. On Azure, federated credentials
remove the secret entirely, which removes the rotation problem rather than scheduling it.
Testing in three layers
Unit tests import from src/ and assert transformation logic against small fixtures. No cluster.
These run on every commit and should complete in under a minute, because a test suite slower than
that stops being run.
Bundle validation is a test. Wire databricks bundle validate for every target into CI.
Integration tests deploy to a dedicated environment and execute against real infrastructure with
synthetic data. Slower and worth it, because they catch what unit tests structurally cannot: a
permission that was never granted, a catalog that does not exist, a cluster policy that rejects your
configuration.
The integration environment should be disposable. If recreating it requires a human, it will drift
from production and its results will stop meaning anything.
flowchart TB
subgraph CI["Continuous integration - every commit"]
direction LR
DEV[Developer] --> PR[Pull Request]
PR --> VAL[Bundle Validation]
VAL --> TEST[Unit Tests]
end
subgraph PROMO["Promotion - gated"]
direction LR
INT[Integration Environment] --> APPR{Approval}
APPR -->|Approved| STG[Staging]
STG --> SMOKE[Smoke Tests]
end
subgraph REL["Release"]
direction LR
PROD[Production] --> EVID[Release Evidence]
end
CI --> PROMO
PROMO --> REL
Alt text: A three-stage delivery pipeline. Continuous integration, running on every commit, goes
from Developer through Pull Request and Bundle Validation to Unit Tests. Promotion, which is gated,
goes from an Integration Environment through an Approval decision to Staging and Smoke Tests.
Release goes from Production to Release Evidence.
Two properties are worth defending in review.
The same artefact moves through every stage. Rebuilding per environment means what you tested is
not what you shipped. The bundle plus a resolved target is what promotes.
Approval is a gate, not a notification. If the approval step can be satisfied by the person who
wrote the change, it is documentation of a process rather than a control.
Rollback, and what it cannot recover
Redeploying the previous bundle version reverts the code and the resource definitions. It does not
revert the data that the bad version already wrote, which is the part that makes rollback for data
systems harder than for stateless services.
A complete rollback procedure covers three things: revert the deployment, decide whether published
output must be withdrawn, and identify who has the authority to make that call.
Delta Lake gives you a mechanism for the data half. RESTORE promotes an old snapshot to be the
current table state, where time travel only reads one:
RESTORE TABLE employee TO VERSION AS OF 1;
RESTORE TABLE employee TO TIMESTAMP AS OF '2026-08-16 00:00:00';
Two retention windows govern this, they default to different values, and the shorter one wins.
delta.logRetentionDuration controls table history and defaults to 30 days. Physical file
retention is governed separately by delta.deletedFileRetentionDuration, which VACUUM honours and
which defaults to 7 days.
So a restore to a point 14 days ago can fail even though the log entry still exists, because the
files it references were already vacuumed. The log entry survives; the files it points to do not.
The documentation puts it directly: running VACUUM costs you the ability to time travel back
further than the data retention period.
Your real rollback window is the file retention setting, not the log retention setting. If your
recovery objective is longer than seven days, raise it deliberately:
ALTER TABLE orders
SET TBLPROPERTIES (delta.deletedFileRetentionDuration = 'interval 30 days');
That aligns file retention with the log default. On Databricks Runtime 18.0 and above,
logRetentionDuration must be greater than or equal to deletedFileRetentionDuration, so raising
file retention beyond 30 days means raising both. Storage costs more; a recovery objective you
cannot meet costs more than that, and you find out at the worst moment.
Drift
Resources changed by hand in the workspace diverge from the definitions in the repository. Someone
adjusts a cluster size during an incident and it stays adjusted.
databricks bundle plan is the command for this. The documentation describes it as building the
bundle and displaying "the actions which will be performed on resources that would be deployed,
without making any changes", so a non-empty plan against production is a drift signal. Run it on a
schedule and alert on one. validate will not do this job: it checks that configuration files are
syntactically correct, which is a different question and the reason both commands exist.
The organisational half is harder than the technical half: the emergency change was probably
correct, and the fix is to bring it back into the repository rather than to revert it.
Release evidence
For each release, keep the commit, the bundle version, the test results, the approval, the deployment
timestamp and identity, and the smoke test outcome. Store it with the code in docs/evidence/.
The obvious reason is audit. The better reason is that six months later, when a table has been wrong
since some point in the past, this is the record that lets you find which release introduced it
without reconstructing history from memory.
Where to start
If your repository shows several of the prototype signals, the sequence I would use:
Extract transformation logic out of notebooks into src/ so it can be tested without a cluster.
That unblocks everything else.
Add a bundle definition covering what already exists, then run validate in CI. You are not
changing behaviour yet, you are making the current state describable.
Split the identities. Move production execution to a service principal.
Add a dev target with mode: development so developers stop testing against shared resources.
Then the promotion path, and only then the drift detection and evidence trail.
Each step is independently valuable, which matters because a migration that only pays off at the end
is a migration that gets abandoned in the middle.
This is an engineering recommendation rather than a Databricks platform guarantee. Verified against
Databricks documentation on 2026-07-29. The bundle behaviours described here have changed before and
will change again; check the current reference before relying on any specific one.