Operating · Chapter 32

CI/CD and Environments

The gap between a data team and a software team has narrowed considerably, and the remaining difference is usually that data teams deploy less rigorously. There is no good reason for that, and the tooling to close it now ships with the platform.

Everything that runs is in a repository #

Notebooks, pipeline definitions, job definitions, SQL, cluster policies, and grants all live in version control, and the deployed state of a workspace is derived from a commit rather than from a person's actions in a browser.

The test is the one in the plain-language box above. An estate where the honest answer involves phrases like "mostly" or "someone would remember" has a documentation problem disguised as a deployment problem, and it usually surfaces as an inability to create a realistic test environment, which is what keeps the test suite thin.

Asset Bundles for what ships with code #

Databricks Asset Bundles describe jobs, pipelines, and the resources an application needs, with per-environment overrides in one file. This is the mechanism we use for anything belonging to a project, while Terraform handles account and workspace topology as described in Accounts and Workspaces.

The division is not arbitrary. Workspace topology changes rarely, is owned by a platform team, and has a blast radius spanning projects. Job and pipeline definitions change constantly, are owned by the team that owns the data product, and should ship on that team's cadence.

# The shape of a bundle: one definition, per-environment overrides
targets:
  dev:
    mode: development
    default: true
    variables:
      catalog: originations_dev
  prd:
    mode: production
    variables:
      catalog: originations_prd
    run_as:
      service_principal_name: sp-originations-prd

Two details in that file do more work than they appear to. The mode: development setting prefixes deployed resources with the developer's name and pauses schedules, so that ten engineers can deploy the same bundle into one workspace without colliding or accidentally running production schedules. The run_as setting means the production deployment executes as a service principal, so the deployment path does not leave with a departing engineer.

A reference topology in code #

It helps to see the division as two bodies of code with two audiences and two release cadences. Terraform owns the account-level estate, which is the set of things that exist before any product does: workspaces, the assignment of a metastore to each of them, the account-level groups that identity federation fills, the cluster policies that constrain what compute may be created, and the catalog grants that decide who reads what. Asset Bundles own what ships with a product, which is jobs, pipelines, and the per-target configuration those need. The sketch below is deliberately partial and is about shapes rather than a working configuration.

# Account-level provider omitted. Resource names are from the
# databricks Terraform provider.

resource "databricks_mws_workspaces" "prd" {
  account_id     = var.account_id
  workspace_name = "tf-prd-analytics"
  aws_region     = var.region

  credentials_id           = databricks_mws_credentials.this.credentials_id
  storage_configuration_id = databricks_mws_storage_configurations.this.storage_configuration_id
}

resource "databricks_metastore_assignment" "prd" {
  metastore_id = databricks_metastore.eu_west.id
  workspace_id = databricks_mws_workspaces.prd.workspace_id
}

resource "databricks_group" "originations_readers" {
  display_name = "originations-readers"
}

# Authoritative for this catalog: grants made by hand are reset.
resource "databricks_grants" "originations_prd" {
  catalog = "originations_prd"

  grant {
    principal  = databricks_group.originations_readers.display_name
    privileges = ["USE_CATALOG", "SELECT"]
  }
}

Three things in that file are worth noticing. The workspace, the metastore assignment, and the group are all account-level objects, so they are applied with account-level credentials rather than workspace ones, which is a different provider configuration and usually a different pipeline. The group is defined once and holds members that arrive from the identity provider rather than from Terraform, so the code owns the group's existence and its entitlements while the identity provider owns its membership. And the grants resource is authoritative for the securable it names, which is the property that makes it valuable and the property that makes it dangerous, as the warning below explains.

The bundle sits in the product repository and describes what the team ships. It complements the target fragment shown earlier in this chapter rather than replacing it, since that fragment showed the target block and this one shows what the targets are overriding.

bundle:
  name: originations

resources:
  jobs:
    daily_load:
      name: originations-daily-load
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ../src/ingest.py

targets:
  dev:
    resources:
      jobs:
        daily_load:
          schedule:
            pause_status: PAUSED
  prd:
    resources:
      jobs:
        daily_load:
          schedule:
            quartz_cron_expression: "0 0 5 * * ?"
            timezone_id: "Europe/London"

The job is defined once and the targets change only what genuinely differs between environments, which here is whether the schedule runs at all. The temptation is to let the two targets drift into two nearly identical job definitions, and the discipline that prevents it is to treat any field appearing under both targets as a candidate for a variable instead.

Who may run these is a boundary worth writing down. Terraform is applied by a pipeline rather than by a person, authenticating as an account-level service principal whose credentials no engineer holds, with a plan produced on the pull request and an apply only on merge to the main branch. Bundle deployments to production likewise run as a service principal, which is what the run_as setting above configures, so that the production deployment path does not belong to whoever last deployed and does not leave when they do. Drift is then a scheduled job rather than a discovery: a nightly plan that fails the build when it is non-empty tells you within a day that production no longer matches the repository, which is the only way to find manual changes before they cause a surprise.

What the pipeline actually runs #

A pull request touching data code should trigger, in order, static checks and formatting, unit tests over transformation logic with no cluster involved, a bundle validation that catches malformed resource definitions before anything is deployed, and then a deployment to an isolated development target where integration tests run against real infrastructure with fixture data.

The unit test layer is the one data teams most often skip and the one that pays back fastest, and it depends on a single structural decision: transformation logic must be ordinary functions that take a DataFrame and return a DataFrame, held separately from the notebook or pipeline file that calls them. Once that separation exists, testing is unremarkable, and the tests run in seconds on a laptop or a build agent rather than waiting for a cluster.

Promotion between environments #

A change moves from development to staging to production as the same artefact with different configuration, never as a separately built thing. The catalog name, source paths, and service principal come from the target's variables, and nothing environment-specific appears in the code.

Staging should hold data that is representative in shape, including the awkward cases, even when it cannot hold real data. A staging environment containing a thousand clean rows validates almost nothing about a pipeline that will meet ten million rows with a null key.

Data changes are deployments too #

A schema change on a published table is an interface change and deserves what interface changes get in software: additive first, a deprecation period for the old shape, and a check for downstream consumers before removal. Unity Catalog lineage makes the consumer question answerable, so we wire it into the review as described in Unity Catalog.

The advanced capabilities worth building toward #

Realistic test data without copying it. A shallow clone of a production Delta table creates a queryable, writable copy that shares the underlying data files, so it costs almost nothing and appears instantly. That makes a per-branch test dataset practical where a full copy never was. On the operational side, Lakebase branching does the same thing for Postgres, which means a migration can be tested against real data on every pull request rather than hoped about.

Pipeline expectations as executable contracts. Declarative pipelines let you assert row-level expectations that fail, drop, or quarantine on breach, as covered in Pipelines. Those assertions run in production continuously, which makes them a test that keeps testing rather than one that passes once in CI.

Environment versions and dependency pinning. Serverless compute exposes an environment version, and pinning it means a platform upgrade does not silently change the behaviour of a running pipeline. Bundles can build and attach a Python wheel, so shared transformation code is versioned and installed rather than copied between notebooks, which is the most common origin of two pipelines that were supposed to agree and quietly do not.

Policy as code. Cluster policies, grants, and budget policies belong in the same repository and the same review as everything else. A CI check that fails when a job definition names an all-purpose cluster, or when a table is granted to all users, catches in review what would otherwise be found in an audit.

Models and agents promoted like code. Models registered in Unity Catalog carry aliases and versions across environments, so promotion is an alias change with lineage rather than a copied artefact, which matters most for the agent workloads in Agents.

Blue-green for anything serving traffic. Serving endpoints support splitting traffic across model versions, which turns a model deployment into a gradual shift you can reverse, rather than a switch you throw.

Rollback #

Delta time travel means a table can be restored to a prior version, which is a real safety net and not a complete one, because restoring one table does not restore the downstream tables computed from the bad data.

We work out the rollback path per pipeline in advance, including which tables are affected and in what order they need reprocessing, and we write it down where the on-call engineer will find it rather than where it was convenient to put it. That document belongs next to the runbooks described in Observability and Reliability.

Table of Contents