Production Databricks Notes: short technical note

A prototype is a legitimate artefact. The problem is a prototype that has quietly been promoted to production by usage rather than by decision, which is how most of them get there.

These seven signals are diagnostic rather than accusatory. Each one is cheap to check and tells you something specific about which failure will arrive first.

1. Everything lives in one notebook

File count is a red herring. The signal is that the notebook has become the unit of deployment, which makes the unit of review a diff of JSON-wrapped cells and the unit of testing a human running it top to bottom.

The question that exposes it: how would you write a unit test for the transformation in cell 14 without executing cells 1 through 13? If the answer requires a cluster, the logic and the orchestration are the same object and neither can be tested alone.

The Databricks-specific version of this is %run. A notebook that pulls in three others through %run has a dependency graph, and that graph exists nowhere you can read it: not in an import statement a linter can follow, not in a call graph an IDE can render, and not in a diff. It exists as magic commands scattered through cells, and the only way to know what a change breaks is to run it. Grep for %run and count. Anything above two is a dependency graph somebody is holding in their head.

The fix is the same one that fixes the rest of this list: move the logic into a package under src/ and import it, so the dependency graph becomes something Python resolves and your tests exercise.

What it predicts: changes get reviewed by reading output rather than by reading code, and a regression ships the first time two people edit adjacent cells.

2. Environment names appear in code

A path, catalog, or workspace URL with dev, staging, or prod in it, written in a source file.

The usual defence is that it is only in one place. It is never only in one place. The reliable test is grep -ri "prod" --include="*.py" --include="*.sql" . and then reading what comes back.

What it predicts: somebody promotes code between environments by editing strings, which means a promotion can be wrong in a way that passes review.

3. A named person owns production execution

Jobs run as ivan@company.com. Everything works until that person changes teams, rotates a credential, or goes on leave, and then the pipeline fails with a permission error that nobody can diagnose because the permission belonged to a human who is not in the room.

There is a second cost that shows up sooner. When workloads run as a person, the audit log cannot distinguish what that person did interactively from what their pipeline did at 03:00. The forensic value of the log drops to near zero at the moment you need it.

What it predicts: an outage whose root cause is an HR event, and an audit trail that cannot answer who changed the data.

4. There are no failure tests

Tests exist. They assert that correct input produces correct output. Nothing asserts what happens on a duplicate, a malformed record, a schema change, or a mid-run failure.

This is the signal I weight most heavily, because the happy path is the part that was manually verified during development anyway. The tests are re-checking the thing you already know works.

The question: which test fails if I remove the deduplication step? In many repositories, none of them do.

What it predicts: the first production incident is in a code path that has never executed successfully or unsuccessfully in a test.

5. Deployment requires manual UI steps

Any step in the release procedure that is performed by clicking. Creating a job, attaching a cluster, setting a permission, uploading a file.

The clicking is cheap. The cost is that the deployed state now exists only in the workspace, so nobody can answer what is deployed by reading the repository, and rebuilding the environment after a loss means reconstructing it from memory.

What it predicts: environments drift apart, and the difference is discovered when something works in staging and fails in production for reasons nobody can enumerate.

A test_customers table in the same schema as customers. A notebook that reads production and writes to a scratch table beside it.

Two failures follow. Somebody queries the wrong one, which is the visible failure. And the permission boundary that should separate exploration from production cannot be drawn, because both live in the same place, which is the failure that matters more.

What it predicts: an access-control design that cannot be tightened later without breaking somebody's workflow.

7. There is no written rollback procedure

Ask what happens if the release you shipped this morning turns out to be wrong. If the answer is a discussion, that is the finding.

Rollback for data systems is harder than for stateless services, because reverting the code does not revert the data it already wrote. That difficulty is the reason to write the procedure in advance rather than the reason to skip it. The procedure needs to cover reverting the deployment, deciding whether published output must be withdrawn, and who has the authority to make that call.

What it predicts: the recovery decision gets made under time pressure by whoever is awake.

Using this

Score a repository out of seven. My rough reading: five or more signals means it is a prototype regardless of what it is currently serving; two or three means it is in transition and the gaps are worth closing deliberately; zero or one means the remaining risk is somewhere other than repository structure.

The order above is roughly the order I would fix them, with one exception. Signal 3, the human identity owning production execution, is worth fixing first even though it looks like a configuration detail. It is usually the cheapest to change and it unblocks the audit and permission work that everything else depends on.

None of these require a specific platform feature to address. They are structural properties of how the project is organised, which is why they survive migrations and re-platforming unchanged.

Companion to the full article on treating Databricks projects as software products, which covers bundle structure, environment targets, deployment identities, and release evidence in depth. This is an engineering recommendation rather than a Databricks platform guarantee.