Start here · Chapter 04

What It Actually Costs

Databricks has a reputation for being expensive. The reputation is partly earned, mostly the result of a comparison that is not like for like, and entirely fixable. This chapter is the honest version.

The unit on the bill #

Databricks bills in DBUs, Databricks Units, consumed per second while compute runs and priced differently by compute type, with interactive all-purpose compute at the top of the range, jobs compute well below it, and SQL warehouses and serverless in between. On the classic compute types you also pay the cloud provider separately for the underlying virtual machines, which is why a Databricks invoice and an AWS invoice both grow when a cluster runs; serverless folds the machines into the DBU rate.

A worked example makes the shape concrete. A 2X-Small SQL warehouse serving an analyst team, running eight hours a day for twenty working days, consumes roughly 160 hours of the smallest warehouse's DBU rate, which at list prices lands in the low hundreds of dollars a month. The same warehouse left running around the clock is 730 hours, four and a half times the cost, for the same amount of work. That multiple, not the rate card, is the story of most surprising bills, and it is why every recommendation in this chapter is about time running rather than about price per unit.

Where the reputation comes from #

Four things, in the order we encounter them.

Idle compute. A cluster left running overnight costs the same as one doing work. This is the single largest line item in most estates we review, and it is entirely avoidable through auto-termination settings that take minutes to configure. Teams coming from a fixed on-premises appliance have no instinct for this, because on an appliance an idle machine cost nothing extra.

All-purpose compute used for scheduled jobs. Interactive compute bills at a higher rate than job compute and is meant for development. A production job pointed at a shared interactive cluster pays that premium every night, quietly.

Nobody could see their own spend. Without tagging, cost is one number arriving at finance, and the team generating it has no feedback loop. People do not optimise what they cannot observe.

A genuinely expensive mistake, left running. A pipeline reading a table with a hundred thousand tiny files, or a dashboard refreshing every five minutes that nobody opens. These are real, and they are bugs rather than properties of the platform.

The comparison people actually make #

Here is the unfair comparison, and it is unfair in both directions.

The point is not that Databricks always wins that comparison, because it does not. For a genuinely simple workload, assembling three AWS services yourself is cheaper and we will tell you so. The point is that a single-service invoice looks expensive next to a set of small invoices that add up to more, and against engineering time that never appears on any invoice at all.

What the traditional clouds actually charge you #

Worth being specific, since the alternative is usually described in the abstract.

Every one of those rows on the right is a service with its own configuration, its own IAM model, its own failure modes, its own monitoring, and its own upgrade path. The integration between them is your code and your problem, and that integration layer is where the real money goes.

Making it cheap, concretely #

Most of the recoverable spend in a typical estate comes from a short list, and none of it requires re-architecting anything.

Attribution before anything else. Every cluster, warehouse, and job carries tags mapping onto cost centre, environment, and product or client, enforced through cluster policies so untagged compute cannot start. Once teams can see their own spend, a meaningful share of waste disappears without anyone being asked.

Aggressive auto-termination. Twenty to thirty minutes on interactive clusters, ten on analyst warehouses. This is usually the largest single saving available.

Serverless for bursty work. It removes idle waste entirely and removes cluster sizing as a thing anyone has to tune. For short and spiky jobs it is now our default.

Job compute for scheduled work. Never a shared all-purpose cluster in a production job definition.

Spot instances for fault-tolerant batch, with an on-demand driver.

Predictive optimisation instead of nightly maintenance jobs that run whether or not a table needs them.

Retire what nobody opens. Query history tells you which dashboards and scheduled queries have no audience, and asking people whether they still need something reliably produces the answer yes.

Where the money genuinely goes #

For planning, the rough shape of a healthy estate is worth knowing. The large majority of spend is compute for scheduled pipelines and interactive analytics. Storage is almost always a rounding error, which surprises people who arrived worried about data volume. Serving endpoints for AI workloads are the line item that behaves least predictably, because their cost scales with usage in a way batch does not, so they get their own budget alerts.

If your estate does not look like that, the shape itself is the diagnosis. Storage dominating usually means retention settings nobody set. Interactive dominating usually means idle clusters rather than busy analysts.

The number that actually decides it #

Compare the fully loaded cost of both paths over three years rather than the invoice over one month, and include the engineer-months. In our experience the crossover is somewhere around the third or fourth data source, which is also roughly where the do-it-yourself pattern stops being simple, as the previous chapter describes.

Below that line we will tell you to stay where you are. Above it, the platform is not the expensive option, it is the one where the expense is visible.

Table of Contents