Operating · Chapter 34

Resilience and Disaster Recovery

This is the chapter every enterprise architecture review turns on, and it is usually where a confident conversation goes quiet. The question is not whether the platform is reliable, because it is, but what your organisation does on the morning a region is unavailable and nobody can say when it will be back.

What the platform already does, and where that stops #

Databricks describes high availability as recovering from an outage affecting a single cloud zone transparently to users, and disaster recovery as recovering from an outage affecting an entire cloud region by continuing in a secondary one. That line is also the line between their responsibility and yours.

Inside a region you get a good deal for free. Core control plane services fail over automatically across availability zones with a documented target of a fifteen minute recovery time and no data loss, backed by a 99.9 percent service level agreement that requires no configuration from you, and serverless compute provides multi-zone failover with nothing to switch on. Classic compute is distributed across zones automatically provided you deployed the workspace subnets into at least two of them, which is a decision made once at deployment time and covered in Accounts and Workspaces. Underneath all of it, the Delta transaction log means an interrupted write leaves a table at its last committed version rather than half updated.

What none of that provides is a second region. A Unity Catalog metastore is regional, workspace secrets are per workspace, and job definitions, dashboard definitions, cluster policies, and grants all live where you created them. A managed disaster recovery capability does exist, replicating catalog metadata, managed table data, and workspace assets with their access control lists into a secondary region, but it is gated behind an application to your account team, it requires the enterprise plan with the mission critical add-on on both workspaces, it leaves the secondary catalogs read-only until failover, and its exclusion list currently omits materialized views, streaming tables, declarative pipelines, secrets, models, vector search indexes, and shares. It is a real reduction in the work rather than an elimination of it.

Our position is that you should assume you own cross-region recovery, and treat any managed capability you qualify for as a shortcut through part of it rather than a replacement for the plan.

One number for the estate is the wrong answer #

Two terms carry the conversation. The recovery point objective is the maximum period of data loss the business can tolerate, and the recovery time objective is the maximum time within which the process must be restored. Both are business decisions rather than technical ones, and the mistake almost everyone makes is setting them once for the platform.

That produces one of two bad outcomes, because either the number is set by the most critical workload and you are now paying to replicate a sandbox catalog at fifteen minute granularity, or it is set by an average and the regulatory report has the same protection as somebody's experiment. We tier instead, and four tiers is usually enough.

The fourth row does more work than the first three, because a meaningful share of most estates does not need replicating at all when the upstream system still has the data and the pipeline that loaded it is in git. We assign every catalog to a tier before writing any replication, since that exercise usually removes half the problem and is what makes the critical tier affordable.

What actually has to exist in the other region #

Data moves either through the cloud provider's own storage replication for raw and external data, or through Delta deep clone for tables, which copies data and metadata and then, on every subsequent run, commits only what changed since the last one. That incremental behaviour is what makes a scheduled clone practical on a table too large to copy nightly.

-- Runs on a schedule in the secondary region, reading the primary storage path.
-- Re-running copies only the commits added since the previous clone.
CREATE OR REPLACE TABLE originations_dr.gold.loan_application_daily
  DEEP CLONE delta.`s3://tf-originations-prd-us-east-1/gold/loan_application_daily`;

The metastore and its grants are re-applied from Terraform rather than copied, which is only true if the grants were defined in Terraform in the first place. An estate where permissions were granted by people in a browser has no recovery path for its permissions, and discovering that during a failover is how a recovered platform ends up open to everyone or open to nobody. Code comes from git, which is the one part almost every organisation already has right.

Jobs and pipelines are redeployed from Asset Bundles into a DR target, following the promotion discipline in CI/CD and Environments. The detail that matters is that the DR target is deployed continuously with its schedules paused, so failover is an activation rather than a deployment. Deploying under pressure into a region you have never deployed into is where most plans discover their gap.

#!/usr/bin/env bash
set -euo pipefail

TARGET=dr

# The same artefact that runs in production, deployed into the DR target.
# That target's schedules are paused, so nothing begins running on deploy.
databricks bundle validate --target "$TARGET"
databricks bundle deploy --target "$TARGET"

# Bring the critical tier up to its last replicated point.
databricks bundle run replicate_tier_one --target "$TARGET"

# Schedules are resumed only after a person has verified the step above.
# That resumption is a deliberate separate action, never part of the deploy.

Secrets have to exist in both regions and their contents legitimately differ, because a connection string in the secondary region points at secondary region resources. This is the item most often missed, and it fails in the least helpful way, which is a pipeline that deploys cleanly and then cannot authenticate to anything.

Then there are the things people forget until a rehearsal finds them, which are streaming checkpoints as covered below, schema registries and any contract store the pipelines read at startup, dashboard and alert definitions that are workspace objects rather than code unless you chose otherwise, service principals and their entitlements, and every external system holding a URL pointing at the primary region, because a recovered platform nobody can reach is not recovered.

Streaming checkpoints, which do not travel the way people assume #

A structured streaming checkpoint is what lets a query restart after a failure, and the instinct is therefore to replicate the checkpoint directory alongside the data and expect the job to resume in the second region. It does not work, and understanding why prevents a bad plan.

The checkpoint is a record of progress against one specific source. It holds the offsets consumed, expressed in terms of that source, which for a Delta source means table versions and for a file source means files already seen at their paths, along with accumulated state for any stateful operation. All of that is meaningful only relative to the exact source it was written against. Point the same checkpoint at a replicated table in another region and the version numbers refer to a different commit history while the file paths refer to a different bucket, so the query is reasoning about progress it never made, and the outcome is either a refusal to start or, worse, a start that quietly skips or reprocesses a window of data.

There are two honest approaches and we have used both. The first is to accept reprocessing from a known-good boundary, which requires the downstream writes to be idempotent, usually through a merge on a business key rather than an append, and it means the recovery plan for a stream is defined in terms of how far back it must go rather than where it left off. The second is to lean on the fact that a deep clone carries stream metadata, so a reader can resume against the clone rather than against a hand-copied checkpoint. Either way this is a design decision made when the pipeline is built, alongside the delivery semantics in Real-Time Ingestion and CDC, rather than a thing discovered on the day.

Warm standby against backup and rebuild #

There are three shapes and the choice trades money against time.

An active-passive warm standby is the common one and the one we recommend for anything in the critical tier. A second workspace exists, infrastructure and job definitions are deployed into it continuously, data replicates on a schedule matching the tier, and nothing runs until failover. It costs the replication, the storage, and the discipline of deploying twice, and it buys a recovery measured in the time a person takes to verify and activate.

Backup and rebuild carries the highest recovery time and the lowest standing cost, and it suits the standard tier. The bet is that Terraform plus bundles plus git can construct a working workspace where none exists, and it is a bet you only win if you have actually run it.

Active-active gives the lowest recovery time and point objectives by running the same workloads in both regions, and it is the most complex and expensive of the three because every job runs twice and the pipelines have to tolerate that. We recommend it only where a workload genuinely cannot be down for an hour.

The ordering is the point. Repointing consumers before verifying the data turns an outage into an incident about wrong numbers, which is considerably harder to recover from than being down, for the reasons set out in Observability and Reliability.

The last row is the argument we make to an architecture review. No platform makes cross-region recovery free, and the difference is how many independent recovery procedures the plan contains, because a plan with nine of them is one nobody rehearses in full.

Testing, which is the whole thing #

An untested DR plan often fails at the moment you need it, and the reasons are always small ones such as a service principal that exists in one region and not the other, a secret scope nobody created, a hard-coded region in a connection string, or a quota in the secondary region that was never raised because nothing ever ran there.

We schedule the exercise rather than intending it, at least twice a year for the critical tier, with a named owner and a runbook that lives in the repository beside the pipelines it recovers. The runbook states, in order, how to declare the event, how to confirm the last replicated position for each critical table, how to deploy and verify before anything is resumed, how to resume schedules, how to repoint consumers, and how to communicate to the people reading the numbers. Failback is written down too and treated as a planned maintenance window rather than an emergency, because synchronising back everything the secondary produced while it was live is genuinely harder than the failover was.

The measure of the exercise is not whether it succeeded, since a rehearsal that succeeds first time usually means the scope was too narrow, but what it found, and every finding should close as a change to code or configuration rather than as a paragraph added to a document. A plan executed against a real secondary region is a capability, and everything else is a paper commitment that the review will accept and the incident will not.

Table of Contents