Start here · Chapter 02
What Databricks Actually Is
Most explanations of Databricks start with the word lakehouse, which is a term invented to describe an architecture rather than to explain one. This chapter describes the platform as the set of jobs it does.
The short version
Databricks is one place to do four things that most companies currently do in four or more separate systems: store data, process it, analyse it, and serve it to people and applications. The argument for it is not that any single one of those is dramatically better than a dedicated tool. It is that they share one security model, one catalog, and one record of where every number came from, so the seams between them stop being your problem. On the traditional clouds those seams are code you write and maintain.
The four jobs #
Everything in this manual sits under one of these, and the parts of the manual are named after them.
Storing is where data lives and who is allowed to see it. On Databricks this is open table formats on your own object storage, governed by Unity Catalog, with an operational Postgres alongside it in Lakebase.
Processing is getting data in and turning it into something trustworthy. Ingestion, pipelines, and orchestration.
Analysing is asking questions, whether in SQL, in a dashboard, in natural language, or by training a model.
Serving is delivering answers to a person, an application, a customer, or another organisation.
What makes it one platform rather than a bundle #
Three things, and they are the whole argument. A vendor could bundle nine services and call it a platform; these are what make it not that.
One catalog over everything. A table, a model, a dashboard, a function, and a volume of files are all objects in the same namespace with the same permission model. Grant a group read on a catalog and that means the same thing whether they arrive through SQL, a notebook, a dashboard, or an agent.
Lineage captured automatically. Because everything runs against the same catalog, the platform records which job wrote which table from which source, down to the column, without anyone instrumenting it. Questions that used to be archaeology become queries, and this is the single feature that most changes how audits and deprecations feel.
One copy of the data. Analytics, machine learning, and streaming read the same bytes under the same governance. No copy means no reconciliation, no second security review, and no divergence.
What "one platform" replaces
| The job | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Governance over everything | Unity Catalog, one model | Lake Formation, IAM, Redshift grants, QuickSight permissions | Purview, Synapse RBAC, Power BI workspace roles | Dataplex, BigQuery IAM, Looker permissions |
| Lineage | Automatic, column level | Partial, per service, mostly manual | Partial, Purview scanners | Partial, Dataplex |
| One copy of the data | Yes | Usually copies between S3, Redshift, and a feature store | Usually copies between ADLS, Synapse, and Power BI import | Usually copies between GCS, BigQuery, and Vertex |
| Batch and streaming in one engine | Yes | Glue plus Kinesis or Flink | Synapse plus Stream Analytics | Dataflow, closest of the three |
| Operational database in the same boundary | Lakebase | Separate RDS estate | Separate Postgres estate | Separate Cloud SQL estate |
| Model training and serving on governed data | Included | SageMaker, separate governance | Azure ML, separate governance | Vertex, separate governance |
The honest note on that table is that each cell on the right is a capable product. Redshift is a good warehouse. BigQuery is an excellent one and in several respects better than what Databricks offers for pure SQL analytics. The comparison is not product against product, it is one governed surface against a set of good products you have to integrate and then keep integrated.
Where the marketing overstates it #
Three claims deserve tempering, because you will hear all three.
Migration is mostly mechanical. Table definitions and straightforward SQL do move mechanically. Stored procedures encoding decades of business rules, scheduling assumptions baked into a legacy orchestrator, and semantics living inside a BI tool's semantic layer do not. Migration is honest about this.
Open formats make you portable. Delta and Iceberg genuinely are open and that matters. A real platform still accumulates dependency on the surrounding managed services, and governance in particular is where lock-in lives. This is a reasonable trade and we make it deliberately rather than pretending it is not a trade.
Everything belongs on the lakehouse. Low-latency transactional serving, high-concurrency operational workloads, and small reference data with tight latency budgets often belong in Lakebase or somewhere else entirely. An architecture that forces every workload onto one substrate is not simpler, it is uniform.
What actually changes for your team #
Three practical consequences, and they are what people report six months in.
Access control stops being per-tool and becomes per-object, centrally. For most organisations this is the largest day-to-day improvement, and it is the subject of Unity Catalog.
Compute becomes elastic, which means engineers make spending decisions directly. That is a feature as long as they can see the consequences, which is what what it actually costs is about.
Machine learning stops being a separate project with a separate copy of the data. Everything in Agents rests on that.