Storing · Chapter 08

Unity Catalog

Unity Catalog is the part of Databricks we care most about getting right, because it is the part that is hardest to change later and the part that determines whether the platform can be trusted with regulated data.

The model in one paragraph #

Objects live in a three-level namespace of catalog.schema.object. A metastore, attached to the account rather than to a workspace, holds the catalogs, while storage credentials and external locations describe where data physically sits and who may reach it. Grants inherit downward, so a privilege on a catalog applies to its schemas and their tables unless something more specific intervenes. Everything readable is lineage-tracked and every access is audited.

The part that surprises people coming from a database background is how much is governed beyond tables. Volumes hold files, so PDFs, images, and model checkpoints get the same permission model as a table, and registered models, functions, and connections to external systems are securable objects too. That is what lets an access review cover the whole platform rather than only the relational part of it.

The credential step is worth pausing on. Users never hold cloud storage permissions themselves; Unity Catalog holds them and issues short-lived, down-scoped credentials for exactly the files a permitted query needs. That is why removing a grant actually removes access rather than removing one of several routes to the same bucket.

How we lay out catalogs #

We use catalogs as the environment and domain boundary, and schemas as the data product boundary. A catalog per environment per domain, named {domain}_{environment}, gives you a namespace meaningful to humans, maps cleanly onto access control, and makes a promotion from staging to production a change of one path segment. Inside a catalog, schemas correspond to the medallion layers or to individual data products depending on the domain's maturity, which is covered in The Medallion Architecture.

Grants belong to groups, never to people #

This is the rule we enforce most aggressively. Every grant is made to an account-level group whose membership is managed in the identity provider, and no user, service principal, or one-off exception ever holds a direct grant. The reason is that direct grants are invisible to the process that governs access. When someone leaves, the identity provider removes them from groups and their access disappears everywhere; a direct grant made during an incident eighteen months ago does not. It is also the difference between an access review that takes an afternoon and one that takes a week. We keep the taxonomy small and predictable, typically a reader, a writer, and an owner group per catalog, with more only where a genuine third pattern exists.

-- The shape of every grant we make
GRANT USE CATALOG ON CATALOG originations_prd TO `originations-prd-readers`;
GRANT USE SCHEMA, SELECT ON SCHEMA originations_prd.gold TO `originations-prd-readers`;

GRANT MODIFY, SELECT ON SCHEMA originations_prd.silver TO `originations-prd-writers`;

Row and column level control #

Unity Catalog supports row filters and column masks as functions attached to a table. Both share a failure mode, which is that the policy becomes undocumented business logic in a SQL function nobody owns. We keep masking functions in the repository with the pipelines, review them like application code, and write tests asserting that a masked reader sees what they should and no more. A masking function without a test is a compliance claim with no evidence behind it.

Beyond the basics #

The features below are where Unity Catalog stops being a permissions system and starts being a governance platform. Attribute-based access control, governed tags and automatic classification have all reached general availability, so on a new estate we now treat them as design decisions to make up front rather than as extras to grow into; the rest of this section is the material most running estates have not yet adopted.

Attribute-based access control. Rather than attaching a filter or mask to each table, you tag columns with governed tags, then write one policy saying any column tagged as an email address is masked for everyone outside a named group. New tables inherit the protection the moment they are tagged, which inverts the usual failure mode where the table created last Tuesday is the one nobody remembered to protect. Pair it with tag policies, which constrain who may apply a tag and what values are allowed.

Lakehouse Federation. Unity Catalog can register a connection to an outside system such as PostgreSQL, SQL Server, Snowflake, Redshift, Synapse, or BigQuery, and expose its tables as a foreign catalog queried in place. The value is not that it removes the need for pipelines, because it does not, but that it lets you put governance and discovery over the whole estate on day one and migrate the data later. That is why a migration as described in Migration can start with governance rather than end with it.

OpenSharing and clean rooms. OpenSharing, previously named Delta Sharing, is an open protocol for giving another organisation live read access to a table without copying it and without them needing a Databricks account, and clean rooms build on it so two parties can compute over combined data with neither seeing the other's rows. For partner data this replaces the file drop and the reconciliation meeting that follows it.

Volumes and service credentials. A managed volume is a governed directory, so landing zones, model artefacts, exported reports and images get the same grants, lineage and audit trail as tables, which matters more every year as unstructured data becomes model input. Service credentials extend the same model to outbound calls, so a job reaching a cloud API uses a governed credential rather than a secret pasted into a notebook.

System tables as the governance surface. The system catalog exposes audit logs, table and column lineage, grants, query history, and billing as ordinary governed tables. An access review becomes a query, so does asking who read a table last quarter, and so does asking what breaks if a column is dropped.

Automatic classification. Databricks can scan tables and propose classification tags for sensitive columns, which then feed the attribute-based policies above. Treat the proposals as an inventory a human confirms rather than as a control in themselves, but as a way to find personal data nobody documented they beat a survey comfortably.

To be fair, Purview is stronger at cataloguing the parts of a Microsoft estate that have nothing to do with analytics, and BigQuery's governance inside BigQuery is excellent. The argument for Unity Catalog is coverage and enforcement in one place, meaning one model that applies whether the access came from a SQL editor, a Python job, a dashboard, or a model, with the audit record for all of them landing in the same queryable table.

What Unity Catalog does not govern #

This boundary matters most for anything agentic. Unity Catalog governs access to data, and it does not govern what a system does with that data once it is read or what business actions it takes in other systems.

An agent with legitimate read access to a customer's loan record that then calls an API to change that loan's terms has violated no Unity Catalog policy, because no data access rule was broken. The action layer needs its own governance, which is the subject of Governed Actions and the reason we built a separate policy evaluation layer rather than trying to express business-action policy in catalog grants.

Lineage is a feature, so use it #

Because lineage is captured automatically for tables, columns, notebooks, jobs, and dashboards, questions that used to require archaeology become queries. We wire lineage into the change process so a pull request touching a gold table lists its downstream consumers automatically, which is the cheapest safety net we know of and takes about a day to build.

Table of Contents