Production Databricks Notes: architecture decisions, investigations, implementation patterns, and lessons for data and AI systems that must operate beyond the demo.

The integration usually starts the same way. A product team needs a number that lives in the lakehouse, so somebody adds a controller action that submits a query, waits for it, and returns the result. It works in development against a warm warehouse and a small dataset.

Then it meets production. The warehouse is cold and the first query takes ninety seconds. A user double-clicks and submits the job twice. A deploy recycles the app mid-request and the caller never learns what happened to the work. The platform returns a rate-limit response and the retry policy turns one problem into several.

None of these are Databricks problems. They are the consequences of modelling a long-running, rate-limited, remotely-executed operation as a synchronous function call. The fix is to stop pretending it is one.

The architectural claim

Databricks operations are asynchronous, long-running, rate-limited, and failure-prone. An HTTP API in front of them should expose an operation resource with its own lifecycle, identity, and persistence, rather than a synchronous endpoint that blocks on a remote system it does not control.

This is the same pattern cloud providers use for their own long-running work, and it is well travelled. What follows is how it applies specifically, and where the interesting decisions are.

sequenceDiagram
    participant Client
    participant API as ASP.NET Core API
    participant Store as Operation Store
    participant DBX as Databricks
    participant Worker

    Client->>API: POST /operations (Idempotency-Key)
    API->>Store: Create pending operation
    API->>Worker: Enqueue operation
    API-->>Client: 202 Accepted + Location

    Worker->>DBX: Submit run or statement
    DBX-->>Worker: External ID
    Worker->>Store: Save external ID

    Client->>API: GET /operations/{id}
    API->>Store: Read status
    API-->>Client: Current state

Alt text: A sequence diagram. The client posts an operation with an idempotency key; the API creates a pending record, enqueues work, and returns 202 Accepted with a Location header. A worker submits the operation to Databricks, receives an external identifier, and stores it. The client polls a GET endpoint which reads status from the operation store.

The property that matters: after the 202, the client and the platform are decoupled. The API can restart, the worker can be replaced, and the operation survives because it lives in your database rather than in the memory of a request handler.

The operation state machine

Give the operation explicit states and make illegal transitions impossible in code. Convention is what you have until someone new joins the team.

stateDiagram-v2
    [*] --> Pending

    Pending --> Submitted
    Pending --> SubmitFailed
    Pending --> Rejected
    Pending --> Cancelled

    Submitted --> Running
    Submitted --> Succeeded
    Submitted --> Failed
    Submitted --> Cancelled

    Running --> Succeeded
    Running --> Failed
    Running --> Cancelled

    SubmitFailed --> Pending : resubmit is safe

    Succeeded --> [*]
    Failed --> [*]
    Cancelled --> [*]
    Rejected --> [*]

    note right of SubmitFailed
        Never reached the platform.
        Safe to resubmit.
    end note

    note right of Failed
        Failed during execution.
        Resubmitting may duplicate work.
    end note

Alt text: An operation state machine. Pending transitions to Submitted, SubmitFailed, Rejected or Cancelled. Submitted transitions to Running, Succeeded, Failed or Cancelled. Running transitions to Succeeded, Failed or Cancelled. SubmitFailed returns to Pending because resubmission is safe when the operation never reached the platform. Succeeded, Failed, Cancelled and Rejected are terminal.

Pending means accepted and durable, nothing sent yet. Submitted means the platform has it and returned an external identifier. SubmitFailed is worth separating from Failed, because an operation that never reached the platform is safe to resubmit and one that failed during execution may not be.

Model it as an enum with a transition guard. The bug this prevents is a late status callback moving a cancelled operation back to running.

public enum OperationState
{
    Pending, Submitted, Running, Succeeded, Failed, SubmitFailed, Cancelled, Rejected
}

public static class OperationTransitions
{
    private static readonly Dictionary<OperationState, OperationState[]> Allowed = new()
    {
        [OperationState.Pending]      = [OperationState.Submitted, OperationState.SubmitFailed, OperationState.Rejected, OperationState.Cancelled],
        [OperationState.Submitted]    = [OperationState.Running, OperationState.Succeeded, OperationState.Failed, OperationState.Cancelled],
        [OperationState.Running]      = [OperationState.Succeeded, OperationState.Failed, OperationState.Cancelled],
        [OperationState.Succeeded]    = [],
        [OperationState.Failed]       = [],
        [OperationState.SubmitFailed] = [OperationState.Pending],   // resubmit is safe
        [OperationState.Cancelled]    = [],
        [OperationState.Rejected]     = [],
    };

    public static bool CanTransition(OperationState from, OperationState to) =>
        Allowed[from].Contains(to);
}

The terminal states have no outgoing transitions, which is the point. A status poll that arrives after cancellation is discarded rather than applied.

Mapping to the platform's states

Your states are yours. The Jobs API has its own life_cycle_state, and the mapping between them is a translation worth writing down explicitly:

Jobs API `life_cycle_state`	Your state
`PENDING`, `QUEUED`, `BLOCKED`	`Submitted`
`RUNNING`, `TERMINATING`, `WAITING_FOR_RETRY`	`Running`
`TERMINATED`	`Succeeded` or `Failed`, per the result state
`SKIPPED`	`Cancelled`
`INTERNAL_ERROR`	`Failed`

Two details make this worth being careful about. A result state is only guaranteed once the run reaches TERMINATED, so reading success or failure from any earlier state is reading a field that may not be populated yet. And Databricks states plainly that "additional states might be introduced in future releases", which means your mapping needs a default branch that treats an unrecognised state as non-terminal. An integration that crashes on an unknown enum value is one platform release away from an outage.

Note also which API version you are targeting. Databricks recommends Jobs API 2.2 for new and existing clients, and the REST reference labels 2.0 and 2.1 the "Jobs (legacy) API". They remain accessible. Paths carry forward under /api/2.2/.

Idempotency, at two layers

Duplicate submission has two independent causes and needs two independent defences.

At your API boundary, accept an Idempotency-Key header. Store it with a uniqueness constraint alongside a hash of the request body. On a repeat, return the original operation instead of creating a second one.

public async Task<IResult> Submit(SubmitRequest request, string idempotencyKey, CancellationToken ct)
{
    var fingerprint = Fingerprint.Of(request);

    var existing = await _store.FindByIdempotencyKeyAsync(idempotencyKey, ct);
    if (existing is not null)
    {
        // Same key, different body means the client has a bug worth surfacing.
        return existing.Fingerprint == fingerprint
            ? Results.Accepted($"/operations/{existing.Id}", existing.ToView())
            : Results.Conflict("Idempotency-Key reused with a different request body.");
    }

    var operation = Operation.Create(request, idempotencyKey, fingerprint);
    await _store.InsertAsync(operation, ct);      // unique index on idempotency_key
    await _queue.EnqueueAsync(operation.Id, ct);

    return Results.Accepted($"/operations/{operation.Id}", operation.ToView());
}

Returning 409 Conflict when the same key arrives with a different body is worth the extra branch. Silently returning the first operation hides a real client defect.

At the platform boundary, the risk is different: your worker submits, the response is lost, and a retry submits the same work again.

The Jobs API supports this directly. Both jobs/run-now and jobs/runs/submit accept an idempotency_token, and the documented guarantee is worth quoting because it is stronger than most such tokens offer:

If a run with the provided token already exists, the request does not create a new run but returns the ID of the existing run instead. [...] Databricks guarantees that exactly one run is launched with that idempotency token. This token must have at most 64 characters.

So derive the token from your operation identifier, which you generated before submitting and which survives a crash. That fits inside 64 characters and makes resubmission safe by construction rather than by careful ordering.

Two caveats. A run deleted after the token was used returns an error rather than launching a new one, so a token is not reusable after cleanup. And no standalone token TTL is published; the practical bound is the 60-day run retention window, after which the associated run is gone.

Record the external identifier anyway. Where an API does not accept a client-supplied token, a narrow window exists between submitting and recording, and a worker that crashes inside it has created an orphan run that nothing will ever reconcile. The sweep that looks for platform runs tagged with an operation identifier your store does not know about is what closes it.

Polling, without hammering the platform

Two polling relationships exist and they need different policies.

Your worker polls Databricks. Use exponential backoff with jitter and a ceiling. Full jitter is the version worth using, because synchronised retries across instances are how a recovering system takes itself down again.

private static TimeSpan NextDelay(int attempt) =>
    TimeSpan.FromMilliseconds(
        Random.Shared.Next(0, (int)Math.Min(30_000, 500 * Math.Pow(2, attempt))));

Clients poll you. Tell them how often. Left to guess they will poll too fast or too slowly, and both become your problem. A Retry-After header on the 202 and on each non-terminal GET converts a guess into a contract, and gives you a lever to widen under load without a client deployment.

If your clients are browsers and the operations are user-facing, server-sent events over the operation resource remove the polling entirely. Keep the polling endpoint anyway, because SSE through corporate proxies is unreliable in ways you will not reproduce locally.

The SQL Statement Execution API has this pattern built in

If your workload is a query rather than a job, the platform already implements a version of the argument this article is making, and its parameters are worth knowing because they let you choose where the waiting happens.

wait_timeout defaults to 10 seconds and accepts 5 to 50 seconds, or 0s for fully asynchronous, which returns a statement ID immediately. on_wait_timeout: CANCEL cancels at the timeout instead of continuing in the background.

That gives you a hybrid: wait briefly, and if the statement finishes inside the window you answer synchronously; otherwise you already hold a statement ID and fall back to the operation pattern. Worth doing, because a large share of queries return inside ten seconds and forcing every caller through a poll loop for those is a cost with no benefit.

Three constraints that shape the design around it:

disposition: INLINE is the default and caps results at 25 MiB in the payload. Above that you need EXTERNAL_LINKS, which returns a short-lived presigned URL in JSON_ARRAY, CSV, or ARROW_STREAM. Design for the second case before a result set grows into it.

You must poll at least once every 15 minutes to keep a statement alive, so a backoff ceiling above that silently kills long-running statements.

Results remain available for one hour after success, and polling does not extend that. If your consumer might not collect within the hour, persist the result yourself. One hour is a collection window, and your consumer's SLA is probably longer than it.

Treat the poll's HTTP status and the statement's state as separate questions. The GET can succeed while the statement has FAILED, so error handling keyed on the status code alone reads a failure as a success. Branch on the state field.

Errors that mean different things

Collapsing platform failures into a single exception type is what makes the retry policy wrong. Three categories:

Transient. Throttling, gateway errors, timeouts, connection resets. Retry with backoff. Honour Retry-After where the platform sends it, because your backoff calculation is a guess and the header is not.

Logical. A malformed query, a missing table, a permission denial. Retrying produces the same failure at cost. Fail the operation and surface the platform's message. Permission denials deserve their own handling because the fix is administrative rather than technical, and an operator seeing "failed" learns less than one seeing "the workload identity lacks SELECT on this table".

Ambiguous. A timeout after submission. You do not know whether the work started. This is the category people forget, and it is the one that causes duplicate execution. Resolve it by querying the platform for the external identifier rather than by resubmitting.

catch (DatabricksApiException ex) when (ex.IsTransient)
{
    await _store.RecordRetryAsync(operation.Id, ex, ct);
    throw;  // let the queue's retry policy handle it
}
catch (DatabricksApiException ex) when (ex.StatusCode is 403 or 401)
{
    await _store.FailAsync(operation.Id, FailureKind.Authorization, ex.Message, ct);
}
catch (DatabricksApiException ex)
{
    await _store.FailAsync(operation.Id, FailureKind.Logical, ex.Message, ct);
}

Identity and tokens

Use a service principal with OAuth machine-to-machine credentials rather than a personal access token. A PAT is bound to a human, inherits their permissions, and expires or dies when they leave. Every argument from the prototype-signals note applies here: a workload authenticating as a person makes the audit log unable to separate what the person did from what the service did.

Databricks says the same thing. The PAT documentation page is titled "Authenticate with Databricks personal access tokens (legacy)", and Databricks recommends OAuth access tokens over PATs for both users and service principals.

The client credentials exchange, against either the account endpoint (https://accounts.cloud.databricks.com/oidc/accounts/<account-id>/v1/token) or the workspace endpoint (https://<instance>/oidc/v1/token):

curl --request POST --url <token-endpoint-URL> \
  --user "$CLIENT_ID:$CLIENT_SECRET" \
  --data 'grant_type=client_credentials&scope=all-apis'

On Azure, prefer workload identity federation so no client secret exists to rotate.

The resulting token is valid for one hour. That number is what makes caching worth building. An uncached integration performs a token exchange on every call, and a naively cached one performs a synchronised refresh across every worker at the same moment.

Cache the token and refresh it before expiry rather than on failure. Refreshing on a 401 means every consumer sees one failed request per token lifetime, and under concurrency it means a thundering herd of refreshes.

public sealed class DatabricksTokenProvider(IMemoryCache cache, IConfidentialClientApplication app)
{
    private static readonly TimeSpan RefreshMargin = TimeSpan.FromMinutes(5);

    public async Task<string> GetAsync(CancellationToken ct)
    {
        if (cache.TryGetValue<CachedToken>(CacheKey, out var cached) &&
            cached!.ExpiresOn - DateTimeOffset.UtcNow > RefreshMargin)
        {
            return cached.Value;
        }

        var result = await app.AcquireTokenForClient(Scopes).ExecuteAsync(ct);
        cache.Set(CacheKey, new CachedToken(result.AccessToken, result.ExpiresOn),
                  result.ExpiresOn - RefreshMargin);
        return result.AccessToken;
    }
}

Under concurrency, wrap the refresh in a per-key semaphore so one caller refreshes and the rest wait.

Correlation across the boundary

Generate a correlation identifier at the edge, store it on the operation, and attach it to whatever Databricks accepts as metadata: job parameters, tags, or a comment on a statement. When an operator is looking at a slow run in the workspace, the identifier is what lets them find the request that caused it.

Use W3C Trace Context so the identifier means something to your existing tooling. With OpenTelemetry, the span for the submit call and the span for the eventual completion belong to the same trace even though they are minutes apart and executed by different processes, which requires storing the trace context on the operation and restoring it in the worker rather than starting a fresh trace.

Instrument four things at minimum: submission latency, queue depth, operation duration split by outcome, and the count of operations in each state. The last one is what tells you the system is stuck, because a queue that is draining and a queue that is not both look busy from the outside.

Rate limiting, in both directions

Databricks enforces limits on its APIs. Your integration should stay under them deliberately rather than discovering them through 429 responses. A concurrency limiter around platform calls, sized below the published limit, converts a platform-side rejection into a local queue you control.

The published per-workspace limits are specific enough to design against, and the asymmetry in them is the useful part:

Endpoint	Limit
`jobs/runs/get`	100 req/s
`jobs/runs/submit`	35 req/s
`jobs/run-now`	20 req/s
`jobs/create`, `jobs/get`, `jobs/list`	20 req/s
`jobs/update`, `jobs/delete`	10 req/s

Plus 2,000 concurrent task runs per workspace, 10,000 job creations per hour, 12,000 saved jobs, and 1,000 tasks per job. Note that anything above 100 tasks requires API 2.2 or later, which is a second reason to target it.

runs/get at 100 req/s against run-now at 20 req/s is what makes the polling design in the previous section affordable: you have roughly five times more budget for asking than for starting. Spend it, and keep the submission path conservative.

These are per workspace, not per application. Your integration is sharing them with every notebook, scheduled job, and other consumer in that workspace, so sizing your limiter at the published number rather than a fraction of it will still produce 429s that are nobody's fault in particular.

Then rate limit your own callers. Without it, one client can consume the entire platform budget and every other consumer of your API sees failures caused by somebody else's loop.

Cancellation that means something

DELETE /operations/{id} should do two things: mark the operation cancelled so no further polling happens, and attempt to cancel the run on the platform. Report both outcomes, because "we stopped watching" and "the work stopped" are different guarantees and the caller is usually paying for the second.

Handle the race where cancellation arrives while the operation is completing. The transition table above makes this safe: Succeeded accepts no transition to Cancelled, so a cancel that loses the race returns the completed result rather than corrupting the record.

Dead letters

An operation that exhausts its retries should land somewhere a human looks, carrying the original request, the correlation identifier, every attempt with its error, and the external identifier if one exists. A dead-letter queue nobody reads has the same problem as a quarantine table nobody reads, which I wrote about in the reliability contract piece: it is a deletion with extra steps.

Alert on the rate of arrival rather than on individual messages. One dead letter is a bad day; a sustained rate is a design problem.

Testing this without a workspace

Unit, contract, and integration. The middle one gets skipped.

Unit tests cover the state machine and the retry classification. No network. These are where you assert that a cancel after success does not corrupt the record.

Contract tests run against a recorded or stubbed Databricks API and assert your client handles each documented response shape, including the ones you hope never to see: 429 with Retry-After, 403, a run that reports failure, a timeout mid-submit. Generate these from the platform's documented responses. What you have personally observed is the happy path plus whatever broke once.

Integration tests run against a real workspace on a schedule rather than per commit. They catch the changes contract tests cannot: a permission that was revoked, a warehouse that was deleted, an API version that moved.

The failure test worth writing first: submit an operation, kill the worker between submission and recording the external identifier, restart, and assert the reconciliation finds the orphan. That sequence is the one that produces duplicate execution in production, and it is invisible to every happy-path test.

What this deliberately does not do

It does not make Databricks operations fast. A cold warehouse is still a cold warehouse. The pattern makes the latency visible and survivable rather than hidden inside a request that may or may not return.

It does not remove the need for a queue. The worker in the diagram is a real background process with its own deployment and failure modes.

It adds a database table, a background worker, and a polling contract to what began as one HTTP call. For a fast operation against a warm warehouse where the caller can tolerate a failure by retrying, the synchronous version is defensible. The threshold I use: if the operation can exceed the client's timeout, or if executing it twice is worse than not executing it, model it as an operation.

This is an engineering recommendation rather than a Databricks platform guarantee. Code is illustrative and simplified for readability. Verified against Databricks documentation on 2026-07-29; exact endpoint shapes and limits change, so check the current API reference before implementing.