Processing · Chapter 16

Orchestration

Orchestration is where a collection of working pipelines becomes a platform that runs unattended, and it is the layer whose design is felt every single day by whoever carries the pager.

What a job is, and what belongs in one #

A Databricks job is a graph of tasks with dependencies between them, where a task can run a notebook, a Python script or wheel, a SQL query, a dbt project, a declarative pipeline, or another job. The graph handles retries, conditional branches, parameter passing, notifications, and triggers, and for the overwhelming majority of data workloads that is sufficient.

The question that matters more than any setting is which dependencies belong in the job graph at all. A dependency between two tables belongs inside a declarative pipeline, which understands table lineage and incremental processing. A dependency between two systems, or between a job and something arriving from outside, belongs in the job graph.

A job graph with forty tasks is almost always a pipeline that escaped into the orchestrator. The shape we aim for is a job per business process, with tasks a non-specialist could name, such as ingest, transform, validate, publish, and notify. When a task cannot be named in one word, it is usually doing two things.

Trigger on data, not on hope #

Time-based schedules encode an assumption about when upstream data arrives, and that assumption decays quietly. A job scheduled for two in the morning because the source used to land at one thirty will keep running at two long after the source moved.

Where the platform supports it we trigger on the event rather than the clock, using file arrival triggers for landing zones and table update triggers for downstream consumers of a table another job writes. Where a clock trigger is genuinely necessary, we pair it with a sensor task that fails fast and loudly if the expected input is absent, rather than letting the job process yesterday's data silently.

The settings that decide whether on-call is bearable #

Retries should have a backoff and a low count, because a job that retries five times with no delay against a source that is down simply fails five times faster. Two or three attempts with a meaningful gap catches the transient case, which is the only case retries help with.

Timeouts belong on every task. The default of running until something gives up means a hung task blocks its dependents indefinitely and burns compute while doing it, and any task without a timeout will eventually meet the pathological input that hangs it.

Concurrency should be capped and the overlap behaviour chosen deliberately. A job that takes longer than its schedule interval will, by default, start overlapping with itself, and two concurrent writers to the same table is a class of bug that is very hard to diagnose after the fact.

Compute, parameters, and environments #

Job compute, created for a run and terminated after it, is the default for scheduled work. It costs less than all-purpose compute, it starts from a clean state, and it cannot be left running by accident. Serverless compute for jobs removes the start-up latency and the capacity planning, and we now reach for it first for short and bursty tasks where cluster start-up was a meaningful share of the runtime. For long-running, heavily tuned workloads with specific instance requirements, a configured job cluster still wins. Shared all-purpose clusters are for interactive development and should not appear in a production job definition at all.

A job definition that mentions originations_prd cannot be tested. We parameterise the catalog, the source paths, and anything else environment-specific, supply the values from the deployment bundle described in CI/CD and Environments, and treat a literal environment name inside a notebook as a review failure.

Beyond the basics #

The features below are the ones teams most often discover a year in, usually during an incident, and each of them removes a piece of machinery people otherwise build by hand.

Repair runs. When a run fails part way through a graph, repairing it re-executes only the failed task and its dependents, reusing the successful output of everything upstream. This is the single most useful feature in the job scheduler and the one people most often do not know exists, because the alternative is re-running a two-hour graph to retry the ten-second task at the end of it.

Task values. A task can publish small values that later tasks read, so a graph can pass a computed watermark, a row count, or a decision downstream without a side table or a file in cloud storage acting as a message queue.

Conditional tasks and run-if rules. An if/else task branches on a condition, and each dependency can specify whether it runs when its parents succeeded, when at least one failed, or regardless. Together these express the two patterns everybody needs, which are skipping expensive work when there is nothing new to process, and always running the notification task even when the run failed.

For-each tasks. A single task can fan out over a list, running the same task body once per input with a configurable concurrency limit. Ingesting forty tables from one source system becomes one task with a list parameter rather than forty near-identical tasks that drift apart over time.

Job composition. A run job task lets one job call another as a unit, so a team can own a job with its own schedule and permissions and still have it invoked as a step inside a wider process. This is how we keep domain ownership intact without building one enormous shared graph.

Health rules and duration alerts. A job can alert when a run exceeds an expected duration rather than only when it fails, which catches the degradation that precedes an outage. A nightly job that has crept from twenty minutes to ninety is worth hearing about before the morning it does not finish.

Git-sourced jobs and queueing. A job can run code directly from a commit in a repository rather than from files in the workspace, which removes the class of incident where someone edited a notebook in production and nobody can say what changed. Separately, a triggered run that arrives while the previous one is still going can be queued rather than dropped, which turns a race condition into an ordered backlog.

The row we would draw attention to is the lineage one. Every other product on that table schedules work perfectly well; none of them can tell you, without additional tooling, which run produced the column a regulator is asking about.

Table of Contents