Start here · Chapter 05

The Vocabulary

Databricks and its competitors use a dense set of terms that mostly describe one of two things: where data lives, and what kind of work you are doing to it. This chapter defines them once so the rest of the manual can use them without stopping.

The short version

Databases come in two shapes. One records what just happened, one transaction at a time, fast, and that is what your application uses; the industry calls that OLTP. The other answers questions across everything that has ever happened, and that is what your reporting uses; that one is OLAP. For forty years those were separate systems, and the pipe between them is where most data engineering effort, cost and error has gone. Every term in this chapter is somebody's attempt to remove that pipe. The lakehouse is Databricks' answer for the reporting side, one governed copy instead of scattered extracts. LTAP, the newest term here, is the attempt to remove the pipe entirely by letting both shapes of work share a single copy of the data.

The two shapes of work #

Everything else follows from this distinction, so it is worth being precise about it.

OLTP means online transaction processing. It describes work that reads or writes a small number of rows identified by key, very quickly, with strict correctness guarantees. Recording a payment, updating a loan status, fetching one customer's profile to render a page. The measure of success is latency per operation and correctness under concurrency. Postgres, MySQL, SQL Server and Oracle are OLTP databases, and so is Lakebase.

OLAP means online analytical processing. It describes work that scans a large number of rows to compute an aggregate. Total originations by region this quarter, the distribution of loan terms, a feature for a model. The measure of success is throughput over volume. Warehouses, Spark, BigQuery, Redshift and Databricks SQL warehouses are OLAP engines.

The reason they are different systems is that they want opposite physical layouts. OLTP stores rows together so that fetching one record touches one place. OLAP stores columns together so that summing one column does not read the ninety columns you did not ask for. That single fact, row storage against columnar storage, is the origin of every pipeline in this manual.

Why one storage layout cannot serve both shapes of work

What you are actually running #

Four more words appear on every page of the platform and deserve plain definitions before anything else does.

A notebook is a document made of runnable cells, mixing code, results and prose. It is where Python and exploratory work happen, the way the SQL Editor is where queries happen. Notebooks are real files with versions and permissions, not scratch space.

Apache Spark is the open source engine underneath the analytical side, created by the founders of Databricks. Its job is to take one query or transformation and spread the work across many machines, which is what makes a scan of a billion rows finish in seconds. You will rarely operate Spark directly on the modern platform, but its name appears everywhere, and "a Spark job" just means work running on that engine.

A cluster is a group of machines running Spark on your behalf, with a driver machine coordinating worker machines. A SQL warehouse is the same idea packaged for queries, sized in t-shirt sizes such as 2X-Small, where a bigger size means more machines working on each query. Serverless means Databricks keeps the machines warm and hands you capacity in seconds when work arrives, so there is nothing to size, start or remember to switch off.

Compute is what you pay for, and it is metered in DBUs, Databricks Units, which are consumed per second while compute runs and billed at a rate that depends on the compute type. The practical consequence, developed in What It Actually Costs, is that idle running compute costs exactly as much as busy compute, which is why switching things off is the first cost lever on the platform.

The attempts to close the gap #

Each term below is a different answer to the same question, which is how to stop maintaining a pipe between the two.

The data warehouse came first. Copy the operational data on a schedule into a system shaped for analysis. This works, and the copy is the problem: it is stale between runs, it has its own security model, and the pipeline that maintains it is code somebody owns forever.

The data lake answered the warehouse's cost and rigidity by putting raw files on cheap object storage. It solved economics and gave up reliability, because a directory of files has no transactions, no schema enforcement and no governance.

The lakehouse is the combination that Databricks pioneered: open table formats such as Delta Lake and Apache Iceberg sitting on object storage, which restore transactions, schema and time travel, plus a governance layer over everything. You get the lake's economics and openness with the warehouse's guarantees. This is the foundation the rest of the platform is built on, and Tables and Storage covers the mechanics.

HTAP meant hybrid transactional/analytical processing, and it was the previous generation's attempt at unification. The approach was to make one engine do both, usually by keeping an in-memory columnar copy alongside the row store. It worked at moderate scale and struggled at large scale, because the two workloads compete for the same machine.

Zero ETL is the current marketing term on the traditional clouds for managed replication between an operational database and an analytical one. It is genuinely useful and it is worth being clear about what it is: the pipeline still exists, it is just operated by the vendor. There are still two copies, two security models, and a replication lag to reason about.

LTAP means lake transactional/analytical processing, and it is Databricks' term for unifying the two at the storage layer rather than in the engine. One copy of the data on open object storage, with transactional and analytical compute reading it independently. LTAP is its own chapter, because it is the newest and least understood part of the platform.

Six answers to the same problem

Approach	What it does	Copies of the data	Who maintains the pipe
Warehouse plus ETL	Scheduled copy into an analytical system	Two	You
Data lake	Raw files on object storage, no guarantees	One, ungoverned	You
Lakehouse	Open table formats plus governance on object storage	One, for analytics	Nobody, for analytics
HTAP	One engine serving both, with an in-memory columnar copy	One logical, two physical	The vendor, inside one machine
Zero ETL	Managed replication between two managed services	Two	The vendor
LTAP	One copy at the storage layer, independent compute	One	Nobody

What people say

OLTP versus OLAP is a distinction from the 1990s. Storage is cheap and engines are fast now, so who cares.

Storage being cheap is exactly why the distinction survived. If storage were expensive, organisations would have been forced to keep one copy decades ago and would have solved this out of necessity. Cheap storage is what made it painless to keep five copies and discover the cost later, in reconciliation meetings rather than on an invoice. The physics has not changed either: fetching one row out of a columnar file still reads more than you asked for, and scanning one column out of a row store still reads the other ninety. Engines got faster at both, which raised the volume at which the difference starts to hurt without removing it. What has genuinely changed is that object storage and open formats got good enough to serve both shapes from one copy, which is the claim in LTAP, and that is a change in the answer rather than a change in the question.

Where the Databricks product names sit #

Now the terms map onto things you can actually turn on.

Unity Catalog is the governance layer. It decides who can see what, records lineage, and covers tables, files, dashboards, models and functions in one model. It is not storage and it is not compute. See Unity Catalog.

Delta Lake and Iceberg are the open table formats. They are what turns files on object storage into tables with transactions.

SQL warehouses are the OLAP compute you point at those tables.

Lakebase is the OLTP side: Postgres, serverless, with its storage separated from its compute, sitting inside the same governance boundary. See Lakebase.

Lakeflow is the data engineering family: Lakeflow Connect for managed ingestion, Spark Declarative Pipelines for transformation, and Jobs for orchestration.

AI/BI is the reporting surface: Dashboards and the conversational Genie, both reading governed tables and shared metric definitions.

MLflow is the model lifecycle, with models registered as objects in Unity Catalog like any other.

How the pieces relate, once the terms are unpacked

The one thing worth remembering #

If you take a single idea from this chapter, take this one. Almost every cost, delay and disagreement in a data platform traces back to having more than one copy of the same fact. Stale dashboards, numbers that do not match, access reviews that take a week, security models that drift apart, and the engineer whose job is keeping a pipeline alive are all symptoms of the same cause.

Every term above is a proposal for how many copies you should tolerate and who should maintain the pipe between them. That is the frame we use when evaluating a platform, and it is why Why Not Just Postgres and a Dashboard? is really a chapter about counting copies rather than a chapter about Postgres.

Table of Contents