Storing · Chapter 12
The Medallion Architecture
Bronze, silver, gold is the most widely repeated pattern in the lakehouse world and also the most widely misapplied, usually because teams adopt the three names without adopting the three contracts that make the names mean anything.
The short version
Data arriving from other systems is rarely fit to make decisions on. The medallion architecture is a convention for handling that in three stages, each with a job. The first stage keeps an exact copy of what arrived, so that when you later discover a mistake you can redo the work rather than asking a source system for history it no longer holds. The second stage turns that raw material into clean, agreed data, where a customer means one thing rather than four things depending on who is asking. The third stage shapes the clean data for the specific uses it has, whether that is a dashboard, a model, or an application. The benefit is not tidiness. It is that when a number is wrong there is one place to look, and when a definition changes there is one place to change it.
What each layer actually promises #
The layers are not stages of cleanliness on a gradient. Each one makes a specific promise to its consumers, and the promise is what determines what is allowed to happen in it.
Bronze promises fidelity. It holds what the source system sent, as it sent it, with ingestion metadata attached and nothing else changed. If the source sent a malformed record, bronze holds the malformed record. The value of bronze is that it lets you reprocess history when you discover that a transformation was wrong, and that value evaporates the moment somebody starts cleaning data on the way in.
Silver promises correctness and conformance. Types are real types, keys are resolved, duplicates are handled, records that fail validation are quarantined rather than dropped silently, and entities from different sources are conformed onto a shared model. Silver is where the business's actual definition of a customer or a loan gets applied.
Gold promises fitness for a purpose. A gold table exists to serve a specific consumer, whether a dashboard, a model, or an API, and it is shaped for that consumer rather than for general use. Gold tables are allowed to be denormalised, redundant with each other, and narrow.
House rules #
These are the rules we apply on every engagement, and the ones we push back on hardest when a client's existing conventions conflict.
Bronze is append-only and never updated in place. Corrections happen by reprocessing into silver, not by mutating history.
Every bronze table carries ingestion metadata as a matter of course, minimally the source file or offset, the ingestion timestamp, and a batch or run identifier. This is what makes a reprocessing question answerable.
Nothing outside the platform team reads bronze. If a consumer needs bronze, that is a signal that silver is missing something, and the fix is to add it to silver rather than to grant access downward.
Validation failures in silver go to a quarantine table with the reason attached, and the quarantine table has an owner and an alert. A pipeline that silently drops six percent of rows will do so for a year before anybody notices.
Gold tables declare their consumer in the table comment. When the consumer goes away, so does the table.
How this maps onto catalogs and schemas #
We use schemas within a domain catalog for the layers, so a table's full name reads as originations_prd.silver.loan_application. This keeps the layer visible in every query and every error message, and it lets grants follow the layer naturally, since the reader group for a domain typically gets SELECT on gold and nothing else. The mechanics of that grant model are covered in Unity Catalog.
For domains that have matured into genuine data products with independent teams and release cycles, we sometimes split gold into a schema per product. That is a scaling decision rather than an architectural one, made when the shared schema gets confusing rather than on principle.
Building the same three layers on each platform
| Concern | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Layer storage | One table format across all three layers | S3 plus Glue Data Catalog, often with Redshift for gold | ADLS plus Synapse, often with a dedicated SQL pool for gold | Cloud Storage for bronze, BigQuery for silver and gold |
| Moving between layers | One declarative pipeline framework spanning all layers | Glue jobs, EMR, or Step Functions per hop | Data Factory or Synapse pipelines per hop | Dataflow or Dataproc into BigQuery |
| Quality rules | Expectations declared in the pipeline, metrics emitted | Glue Data Quality on Glue tables | Data Factory data flow assertions | Dataplex quality scans, configured separately |
| Governance across layers | One permission model over every layer | Lake Formation for the lake, Redshift grants for the warehouse | Purview plus Synapse plus storage ACLs | BigQuery IAM plus Cloud Storage IAM |
| Lineage across layers | Captured automatically, column level | Glue lineage in DataZone, partial | Purview lineage, partial for custom code | Dataplex lineage, partial |
| Replay from bronze | Time travel on the same table | Rebuild from S3, no table history unless you added Iceberg or Hudi | Rebuild from ADLS, same caveat | Rebuild from Cloud Storage, same caveat |
The other platforms can plainly build a medallion architecture, and many organisations have. The difference is where the seams fall. On the cloud-native stacks the layers usually sit in different services with different permission models and different lineage stories, so the boundaries between layers are also boundaries between products, and keeping governance consistent across them is ongoing work somebody owns. On Databricks the layers are schemas in one catalog with one permission model, and the boundaries are conventions you enforce rather than integrations you maintain.
Beyond the basics #
Most teams stop at the three names. The capabilities below are what make the pattern cheap to operate rather than merely tidy, and in our experience they are the least known part of the platform.
Replay is a design property, not a hope. Because every layer is a Delta table with a transaction log, you can rebuild a downstream table from an exact upstream version rather than from whatever the upstream happens to look like now. Reading bronze.events VERSION AS OF 41822 in a rebuild makes the result reproducible, which turns "we reran it and got a different number" into a solvable problem. Bronze retention properties, delta.logRetentionDuration and delta.deletedFileRetentionDuration, are what determine how far back that ability extends, so they are worth setting explicitly per layer rather than inheriting.
Change data feed moves only what changed. Enabling delta.enableChangeDataFeed on a silver table lets gold consume inserts, updates, and deletes as a stream rather than recomputing an aggregate over the whole table. On a wide silver table feeding several gold aggregates this is frequently the single largest compute saving available, and it is one line of table properties.
Deletions have to cross the layers. An erasure request under GDPR or a similar regime is where bronze fidelity and legal obligation pull against each other. The practical approach is to keep the identifier that links a subject across layers, delete in silver and gold, and either delete or pseudonymise in bronze depending on your legal position. Deletion vectors make the delete cheap, since rows are marked rather than rewritten. What matters is that the procedure exists and has been tested before somebody asks for it.
Governance at the boundary, not more tables. Teams often fork a gold table into three variants because three audiences see different subsets. Row filters and column masks in Unity Catalog let one table serve all three, applied at query time by group membership, and one filtered table is easier to keep correct than three that drift.
Gold does not have to stay inside the platform. A gold table can be shared to another organisation through OpenSharing without copying it, synchronised into Lakebase for low-latency application reads, or served to a model endpoint. Treating gold as the publication boundary rather than the last stop is what makes the rest of the estate simple, and the consumption side of that is covered in Consumption.
The layers want different physical treatment. Bronze is append-heavy and read rarely, so it wants large files and long retention. Silver is merge-heavy and benefits most from liquid clustering on the join and filter keys. Gold is read constantly by dashboards and wants clustering on the columns those dashboards filter on. Predictive optimisation handles the maintenance, as covered in Tables and Storage.
When not to use three layers #
A domain with one source, no conformance work, and a single consumer does not need three layers, and forcing them produces two tables that are copies of each other plus a maintenance burden. We are comfortable with a two-layer domain as long as the fidelity promise of bronze is preserved somewhere.
Equally, some domains need a fourth concept, usually a reference or dimension schema shared across layers, or a features schema serving model training. Adding one is fine. What is not fine is having the same table exist twice under different names because two teams disagreed about which layer it belonged in.