Operating · Chapter 33
Observability and Reliability
A data platform fails differently from an application. Applications tend to fail loudly, returning errors somebody notices, while data platforms fail quietly, producing a number that is wrong in a way nobody can see until a person makes a decision on it.
The short version
The dangerous failure on a data platform is not the one that pages someone at three in the morning. It is the one where everything reports success and the numbers are wrong, because a source system quietly changed the meaning of a field, or a file never arrived, or a job that was supposed to run simply did not start and therefore never failed. Monitoring that only watches whether jobs succeeded will not see any of that. What catches it is watching the data itself: is it as fresh as people believe, did roughly the expected amount of it arrive, and do the values still look like they looked last week. That shift, from watching the machinery to watching the output, is most of what separates a platform people trust from one they quietly work around.
The four things worth monitoring #
Everything we alert on falls into one of four categories, and teams reliably instrument the first and neglect the other three.
Job health, meaning did the run succeed, how long did it take, and did it consume what it usually consumes. Runtime and cost trends matter as much as success, since a job that has quietly doubled in duration over a month is telling you something before it starts failing.
Freshness, meaning is the data as current as consumers believe it is. This is the check that catches a job which never started, a failure mode that produces no failure notification at all. We define a freshness expectation per published table and alert on breach, because that single check covers both the crash and the silent non-start.
Quality, meaning are the pipeline expectations passing, is the quarantine table growing, are null rates and distributions where they were yesterday. Distribution monitoring is what catches an upstream system changing the meaning of a field, which is the failure mode that produces confidently wrong numbers rather than errors.
Volume, meaning did roughly the expected quantity of data arrive. A source delivering ten percent of its usual rows is a problem no schema validation will catch.
Alert on consumer-facing symptoms #
The distinction that keeps an on-call rotation sustainable is between things needing someone now and things needing someone tomorrow.
A gold table feeding a regulatory report going stale is an incident. A bronze ingestion job failing its first retry is not, provided the retry succeeds and the downstream freshness expectation holds. Alerting on both at the same severity trains people to ignore the channel, and the cost of that training is paid on the day something real happens.
We route quality warnings, quarantine growth, and non-blocking failures to a review queue somebody works through on a schedule, and we reserve paging for a breach of something a consumer depends on.
Runbooks, and where they live #
Every alert links to a runbook, and a runbook that does not say what to check first is not a runbook. The minimum useful content is what this alert means, which consumers are affected, what to check in what order, what the safe corrective actions are, and when to escalate.
Teams usually put them in a wiki nobody opens during an incident. We keep them in the repository next to the pipeline they describe, so they are versioned with the thing they document and reviewed when it changes, which follows from the practice in CI/CD and Environments.
Incident practice specific to data #
Two habits make data incidents shorter.
Establish the blast radius before fixing anything. The question is which downstream tables, dashboards, models, and external consumers touched the bad data and over what window, and Unity Catalog lineage answers it in minutes, whereas the instinct is to start correcting the source table immediately.
Communicate that a number was wrong even after it has been fixed. A dashboard that was wrong for six hours was used to make decisions during those six hours, and the people who made them need to know. Data incidents have this property that application incidents mostly do not, and skipping the communication is how a platform loses trust that takes a year to rebuild.
The advanced surface most teams never switch on #
The basics above can be built with scheduled queries and alerts. The capabilities below are already in the platform, and most estates we inherit are using a fraction of them.
System tables are the foundation, and they change the character of the whole exercise because operational telemetry is queryable data rather than logs in another tool. Job run history, query history, billable usage, audit events, and lineage are all tables, which means your monitoring is SQL you already know how to write, joined across dimensions that would live in four separate products elsewhere. The question "which team's queries got slower after the change, and what did that cost" is one query here and a project on an assembled stack.
Lakehouse Monitoring profiles a table on a schedule and produces metrics tables plus a generated dashboard, tracking distributions, null and zero rates, and drift against a baseline or against the previous window. This is the mechanised version of the quality and volume checks above, and its inference profile also tracks model inputs and predictions over time, which is how you notice a model degrading before the business does.
Job health rules and duration thresholds let a job alert on running longer than expected or on a streaming backlog exceeding a bound, rather than only on failure. That is the difference between finding out a job is degrading and finding out it has stopped.
The pipeline event log exposes every declarative pipeline run as a queryable table, including expectation pass and fail counts per batch. Trending those counts turns data quality from a pass or fail moment into a series you can watch move.
Delta table history records the operation, the version, and the row counts for every write, which makes "what changed this table and when" answerable after the fact without any monitoring having been configured in advance. During an incident this is frequently the fastest route to the cause.
Inference tables and MLflow tracing capture requests, responses, and intermediate steps from serving endpoints and agents, which is the only practical way to debug the workloads described in Agents, where the failure is usually a bad answer rather than an error.
Log and metric delivery outward keeps the platform's audit and operational data flowing into the central estate, so that data incidents appear on the same timeline as everything else during a cross-system investigation.
Assembling the same picture on each platform
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Operational telemetry as queryable tables | System tables, plain SQL | CloudWatch Logs Insights plus CloudTrail in Athena | Monitor plus Log Analytics with KQL | Cloud Logging sinks into BigQuery |
| Data quality and drift monitoring | Lakehouse Monitoring, built in | Glue Data Quality, per job | Purview data quality, separate service | Dataplex data quality |
| Freshness expectations on a table | Native alerts on tables and pipelines | Build it with EventBridge and Lambda | Build it with Data Factory and Monitor | Build it with Composer and Cloud Monitoring |
| Column-level lineage for blast radius | Unity Catalog, automatic | Glue lineage, partial | Purview, scan-based | Dataplex, partial |
| Cost and performance in the same query | Yes, billing joins to query history | Cost Explorer separate from logs | Cost Management separate from Monitor | Billing export separate from logs |
| Model and agent request tracing | Inference tables and MLflow tracing | SageMaker Model Monitor | Azure ML monitoring | Vertex AI model monitoring |
Retrospectives that change something #
The retrospective question producing improvement is not why the pipeline broke, since the answer is usually a specific upstream event that will not recur in the same form. It is why nobody knew for four hours, and the answer to that is almost always a missing check of the freshness, volume, or distribution kind. Each incident should end with a specific monitor added, and if it cannot, the incident is likely to happen again in a different costume.