Reference · Chapter 35
Databricks Versus the Traditional Clouds
Every chapter in this manual compares its own subject against the native services. This chapter puts the whole comparison in one place, so that an architecture review can start from a single page.
The short version
You can build everything Databricks does out of services you already have on AWS, Azure or GCP. Nobody serious disputes that. The question is what assembling it costs, and the answer has three parts: the integration code between eight or nine services, the governance model you have to invent because no single service spans them, and the team you need to keep both working as each service changes underneath you. Databricks is the argument that this integration is a solved problem worth buying rather than a differentiator worth building. Where a workload needs only two or three of those services and always will, the traditional clouds are the better answer and we will say so.
The whole platform, side by side #
The left side is one product with one permission model. The right side is a set of good products, each with its own permission model, connected by code you own. That picture is the entire argument, and everything below is detail.
Storing #
Where the data lives and who may see it
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Open table format | Delta and Iceberg, first class | Iceberg on S3, engine support varies | Delta on ADLS, Synapse-centric | Iceberg and native BigQuery storage |
| Catalog | Unity Catalog, spans every object type | Glue Data Catalog, tables only | Purview plus per-service metadata | Dataplex plus BigQuery |
| Fine-grained access | Row filters, column masks, ABAC from tags | Lake Formation, plus separate Redshift and QuickSight grants | Synapse RBAC plus Power BI roles plus Purview | BigQuery policy tags plus per-service IAM |
| Lineage | Automatic, column level, across jobs and dashboards | Manual, or partial per service | Purview scanners, partial | Dataplex, partial |
| Operational Postgres in the same boundary | Lakebase, with branching | RDS or Aurora, separate governance | Azure Database for PostgreSQL, separate | Cloud SQL, separate |
| Copy production database instantly | Branch in about a second | Snapshot and restore, minutes to hours | Same | Same |
| Cross-organisation sharing | Delta Sharing, no copy | Data Exchange, or copy to the recipient | Data Share | Analytics Hub |
The row worth arguing over is fine-grained access. On the traditional clouds, "who can see this customer's data" has a different answer in the lake, in the warehouse, and in the BI tool, and keeping those three in agreement is a permanent manual job. That is not a gap in any one product, it is what happens when three products each own part of the answer.
Processing #
Getting data in and turning it into something trustworthy
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Incremental file ingestion | Auto Loader, state managed for you | Glue bookmarks, or your own S3 listing | Data Factory with your own watermark | Dataflow with your own state |
| Managed source connectors | Lakeflow Connect | AppFlow plus DMS, partial coverage | Data Factory connectors, broad | Datastream, narrower |
| Batch and streaming in one engine | Yes, same code | Glue plus Kinesis or Managed Flink | Synapse Spark plus Stream Analytics | Dataflow, genuinely unified |
| Declarative pipelines with quality rules | Lakeflow Declarative Pipelines, expectations built in | Build it with Glue plus Deequ | Build it with Data Factory plus custom checks | Build it with Dataflow plus custom checks |
| Data quality as a first-class metric | Expectations, quarantine, event log | Bolt on | Bolt on | Bolt on |
| Orchestration aware of table lineage | Jobs, with table update triggers | Step Functions or MWAA, lineage-blind | Data Factory, lineage-blind | Cloud Composer, lineage-blind |
GCP deserves credit here. Dataflow is a genuinely unified batch and streaming engine and in some respects a cleaner model than anything else on the list. The gap is not the engine, it is that Dataflow has no opinion about data quality, no declarative pipeline layer, and no shared catalog with the rest of the estate.
Analysing and serving #
Answering questions and delivering the answers
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Interactive SQL | Serverless SQL warehouses | Athena, or Redshift Serverless | Synapse Serverless SQL | BigQuery, best in class |
| Dashboards | Built in, on governed tables | QuickSight | Power BI, best in class | Looker |
| Natural language over governed data | Included, reads catalog metadata | Q, separate configuration | Copilot in Power BI | Gemini in BigQuery |
| Model training and registry | MLflow, models as catalog objects | SageMaker, separate governance | Azure ML, separate governance | Vertex AI, separate governance |
| Model serving on governed data | Included, inference tables for evaluation | SageMaker endpoints | Azure ML endpoints | Vertex endpoints |
| Vector search synced to source tables | Managed sync | OpenSearch or Aurora pgvector, sync is yours | AI Search, sync is yours | Vertex Vector Search, sync is yours |
| Low-latency application reads | Lakebase, fed from gold | RDS replica plus ElastiCache plus your API | Same shape | Same shape |
Two honest concessions on this table. BigQuery is an outstanding SQL engine and for a pure SQL analytics workload with no pipeline, governance, or ML requirement, it is a better product than a Databricks SQL warehouse. Power BI is the strongest BI tool of the group and its Databricks connector is good, so being a Power BI organisation is not a reason to avoid the platform.
Operating #
Running it once it exists
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Cost attribution | Tags and budget policies into billing system tables | Cost Explorer plus per-service tagging | Cost Management | Billing export to BigQuery |
| Audit across the whole estate | One audit log covering every object type | CloudTrail plus per-service logs | Monitor plus per-service logs | Cloud Logging plus per-service |
| Deployment | Asset Bundles plus Terraform | CloudFormation or CDK per service | Bicep or ARM per service | Deployment Manager or Terraform |
| Identity | Federated once at the account | IAM plus per-service grants | Entra ID plus per-service RBAC | Cloud IAM plus per-service |
| Freshness and quality monitoring | Lakehouse Monitoring, built in | Build it | Build it | Build it |
| Services to patch and upgrade | One platform | Eight or nine | Eight or nine | Eight or nine |
Where the traditional clouds genuinely win #
We would rather write this section than have you find it yourself and conclude the rest was marketing.
A single simple workload. If what you need is to land files in object storage and query them with SQL, then S3 with Athena, or ADLS with Synapse Serverless, or GCS with BigQuery, is cheaper, simpler, and entirely sufficient. Adding a platform there adds moving parts and removes none.
Pure SQL analytics at scale. BigQuery is excellent and its serverless model is genuinely frictionless. If the whole requirement is a warehouse, that is a strong answer.
Deep integration with one cloud's application services. If your architecture is built around Lambda, Step Functions, and DynamoDB, and the data work is a small part of it, staying inside that world has real value in operational coherence.
An existing, well-run platform team. Some organisations have already built the integration layer, staffed it, and run it well. Replacing something that works is rarely a good use of a year.
Sovereignty or regional availability constraints. Availability differs by region and by cloud, and occasionally the native services are the only ones present where the data must stay.
How to run the evaluation #
Comparing feature lists produces a stalemate, because most rows are ties and the ones that are not get argued about. Compare work instead.
Take one real requirement that spans at least three sources and ends in something a customer or an executive sees. Have your team estimate it on your current stack honestly, including the governance and monitoring work, not just the happy path. Then estimate it on Databricks. The gap is almost never in the transformation logic, which is similar on both, and almost always in the integration, the access model, and the monitoring.
Then take one operation your team currently avoids because it is expensive, such as copying production data for a test, reprocessing a quarter of history, or granting a new team access to a subset of a table. Time it on both. Capabilities that are technically possible but operationally expensive do not change how people work, and the platform argument lives entirely in that difference.