Operating · Chapter 30

Cost and Performance

Elastic compute means engineers make spending decisions dozens of times a day, usually without realising it. This chapter is about the mechanics of making those decisions visible and making slow things fast, in the order that actually works.

The business case for the platform, and the "Databricks is expensive" objection, are covered in What It Actually Costs. This chapter assumes you have already bought the thing and now have to run it well.

Performance, in the order to check things #

When something is slow the causes cluster into a short list, and the order in which you check them is remarkably reliable. Working through it in order takes less time than guessing, and it stops you from arriving at the answer that is always available and usually wrong.

File layout comes first. Small files are the most common cause of slowness and the one least likely to be suspected. A table accumulating thousands of tiny files from streaming writes or frequent small merges will be slow no matter what compute you point at it, because the engine spends its time opening files rather than reading rows. The fix is compaction and a sensible clustering key, not more nodes.

Then skew. A single key value holding a disproportionate share of the rows produces the signature symptom of one task still running long after every other task in the stage has finished. The query profile shows it immediately once you know to look at the distribution of task durations rather than at the total.

Then statistics. The optimiser chooses join strategies from statistics, and stale or missing statistics produce plans that look inexplicable, most often a shuffle join where a broadcast would have been correct.

Then caching. A repeated identical query against unchanged data should be served from the result cache. A workload that never hits cache usually contains a non-deterministic element, and the usual culprit is a timestamp function in the predicate that makes every execution technically a different query.

Only then sizing. By the time you get here it usually is not the answer, and if it is, you now know that rather than assuming it.

Choosing the right compute #

Most estates we review are paying a premium for using the wrong kind of compute rather than too much of it.

All-purpose compute is interactive, bills at the higher rate, and exists for development, so a production job pointed at a shared all-purpose cluster pays that premium every night. Job compute spins up for a run and disappears afterwards, which is what scheduled work should use. Serverless removes the sizing question and the idle question together, and for short, bursty, or unpredictable work it is now our default. SQL warehouses serve analytics and BI, and the serverless variant starts in seconds rather than minutes, which matters more than it sounds because a warehouse taking four minutes to wake gets left running all day.

Warehouse sizing works on two separate axes that teams routinely confuse. The t-shirt size determines how much a single query can parallelise, so it addresses one big slow query. The maximum cluster count determines how many concurrent queries can be served before queuing, so it addresses many users at once. Increasing the size to fix a queuing problem, or adding clusters to fix a single slow scan, spends money without changing anything.

For batch work that tolerates interruption, spot or preemptible workers with an on-demand driver cut the compute bill substantially and confine the risk to work that can simply be retried.

Attribution, and how it actually gets enforced #

The first thing we build on a new estate is attribution, because every other cost control depends on it.

Every cluster, warehouse, job, and serverless workload carries tags mapping onto cost centre, environment, and product or client. The enforcement mechanism is cluster policies, which can require a tag, constrain its allowed values, and refuse to start compute that lacks it. Serverless work is attributed through budget policies, which attach the same tags to workloads that have no cluster definition for a tag to live on. Those tags then flow into the billing system tables, where they become the dimension every cost query groups by.

The system tables are the part people underuse. They hold billable usage by workload, tag, and SKU, list pricing so you can compute currency rather than units, plus query and job run history, so you can join spend to the specific query that caused it. That join turns a cost conversation from an argument into a list.

Budgets and alerts #

We set budget alerts per workspace and per major tag, with thresholds that fire at a fraction of the period rather than at the end of it, because an alert that fires when the money is gone is a report. Serving endpoints for AI workloads get their own alerts, since their cost scales with usage in a way batch simply does not.

The advanced surface most teams never touch #

Everything above is the basics. The capabilities below are where the larger gains sit, and most estates we inherit are using none of them.

Liquid clustering replaces static partitioning and Z-ordering with a clustering scheme you can change without rewriting the table. Partitioning decisions are usually made early, on incomplete knowledge of the query patterns, and then become permanent because changing them means a rewrite. Liquid clustering makes that decision reversible, and the automatic variant chooses and adjusts the keys from observed query patterns.

Predictive optimisation runs compaction, clustering, and vacuum when the data and query patterns indicate they are needed, rather than on a nightly schedule that runs whether or not a table has changed. Hand-maintained maintenance jobs almost always run too often on tables that do not need it and not often enough on the one table that does.

Deletion vectors turn a delete or update from a rewrite of every affected file into a marked-row operation, with the physical rewrite deferred to maintenance. On tables with frequent small corrections this changes merge performance by an order of magnitude.

Photon is the vectorised execution engine, and it changes the arithmetic rather than simply making things faster. It bills at a higher rate per unit while typically completing the work in less time, so for scan-heavy and aggregation-heavy SQL it is usually cheaper in total and for some workloads it is not. This is worth measuring on your own queries rather than assuming either way.

Adaptive query execution re-plans a query mid-flight using actual statistics from completed stages, which is what handles many skew and join-strategy problems without intervention. Knowing it exists tells you why the plan you read in the UI is not always the plan that ran.

The query profile is the single most useful diagnostic tool on the platform and the least opened. It shows time spent per stage, bytes and files read against bytes pruned, spill to disk, and the task duration distribution that exposes skew. Reading one profile properly usually ends the debate about what is slow.

Instance pools keep warm instances available so that job clusters start in seconds rather than minutes, which matters for short frequent jobs where startup time can exceed the work.

Materialised views and streaming tables refresh incrementally, computing only what changed. Replacing a full nightly rebuild with an incremental refresh is often the largest single saving available on a mature estate, and it is a change to the definition rather than to the architecture.

The pattern underneath all of this is the one that also runs through Observability and Reliability, which is that you cannot fix what you cannot see. Put the measurement in place first, then let the people generating the number look at it.

Table of Contents