Analysing and serving · Chapter 22

Sharing, Clean Rooms, and Marketplace

Almost every data platform eventually has to hand something to somebody outside the organisation, and the way that usually happens is an extract, a file, and an agreement nobody can enforce once the file has left. There is a better set of options now, and choosing between them is mostly a question of how much the other party is allowed to see.

Sharing without copying #

The sharing feature is now called OpenSharing, and it was previously known as Delta Sharing, which is the name most search results and most of your team still use. The protocol is open, and it is also the substrate underneath both Clean Rooms and Marketplace, so understanding it is worth more than the sharing use case alone.

Three objects carry the model. A provider is the entity that shares data. A recipient is the entity that receives it. A share is a read-only collection of tables and table partitions that lives in the provider's metastore, and it is the unit you grant. The recipient reads the provider's actual storage through short-lived pre-signed URLs rather than receiving a copy, which is the property that makes the whole thing worth the effort, because a partner reading your gold table today sees the version you published this morning without any pipeline on their side and without any pipeline on yours.

What can go into a share depends on which mode you are in, and this is the first real decision.

The two modes, and what each one costs you #

Databricks-to-Databricks sharing applies when the recipient's workspace is attached to a different Unity Catalog metastore from yours, including one in another cloud or another region. You identify them by their sharing identifier rather than issuing a token, there is no credential to rotate or lose, and the share arrives on their side as a catalog they can grant on internally. This mode carries the richer asset set, covering tables, streaming tables, managed Iceberg tables, views, materialized views, Unity Catalog volumes for non-tabular files, Unity Catalog models, and notebook files. Column comments and primary key constraints travel with the table, which matters more than it sounds when the recipient is trying to understand what they have been given.

Open sharing applies when the recipient is on anything else, meaning a partner running pandas, a data science team on another platform, or a business intelligence tool with a sharing connector. Here the recipient is authenticated by a bearer token delivered in a credential file, or through OIDC federation where their identity provider can be involved, and the asset set narrows to tabular data in Delta or managed Iceberg format. It is genuinely open in the sense that the recipient needs no Databricks account at all.

The security posture differs in one respect that governs how we advise clients. A credential file is a bearer secret, so it is exactly as safe as the recipient's handling of it, and a file mailed to a partner and left in a shared drive is a durable liability. The Databricks-to-Databricks mode has no such artefact, because the trust is anchored in a metastore identity rather than in a string. Where the recipient is on Databricks we use that mode without discussion, and where they are not we treat the credential file as a production secret with an owner, a rotation schedule, and an IP access list narrowing where it can be used from.

-- Run in the provider metastore that owns the data
CREATE SHARE IF NOT EXISTS originations_partner_share
  COMMENT 'Daily gold originations aggregates for dealer group partners';

-- WITH HISTORY lets the recipient time travel and read the table as a stream
ALTER SHARE originations_partner_share
  ADD TABLE originations_prd.gold.loan_application_daily
  AS partner_gold.loan_application_daily
  WITH HISTORY;

-- A recipient on another Unity Catalog metastore, identified rather than tokenised
CREATE RECIPIENT IF NOT EXISTS northwest_motors
  USING ID 'aws:us-east-1:19a84bee-54c1-4f0e-9b21-6f7bd4c0a112'
  COMMENT 'Dealer group partner, contract DG-2031';

GRANT SELECT ON SHARE originations_partner_share TO RECIPIENT northwest_motors;

Adding a whole schema rather than named tables is the option to reach for when the partner relationship is broad, because it carries the tables, views, models, and volumes in that schema along with anything added to it later. That convenience is also the reason we do not use it on a schema anyone can write to, since a table created by a colleague on a Tuesday afternoon becomes a shared table without a review having happened.

Revocation and audit, which is the part that sells it #

Access can be withdrawn on demand and at more than one level of granularity, so you can revoke a token, deny a specific share to a specific recipient, or restrict the addresses a recipient may connect from. The important property is that revocation is immediate and complete, because the recipient never held the data, only the right to read it.

On the audit side, share creation and recipient activity land in the audit log and in system tables, which means the questions a contract review actually asks become SQL. Who has been granted this share, when did they last read it, which tables did they touch, and what happened after we revoked. Those are the same system tables the practice in Observability and Reliability already draws on, so the monitoring is an extension of what you have rather than a separate exercise.

Clean rooms, and what they actually guarantee #

Sharing answers the case where one party may see the other's data. A clean room answers the case where neither may.

The guarantee is narrower and more precise than the marketing usually suggests. Two or more parties each share assets into an isolated, ephemeral central environment that Databricks manages, using OpenSharing to get them there, and the collaborators can see column names and column types but not each other's rows. Work happens through notebooks, and the mechanism that makes the guarantee real is that a notebook must be approved by the other collaborators before it can run, with only the most recent version runnable. All collaborators hold equal privileges, including the party who created the room, and once the room exists it is locked down so that no new collaborator can join. If any party removes the clean room, it is void for everyone.

Tables, views, volumes, and notebooks can be brought in, and results come back as output tables. Serverless compute and Unity Catalog are prerequisites, egress from the room is restricted, and activity is recorded in a clean room events system table and the audit log.

The limits deserve stating plainly, because a clean room is a strong control that is easy to over-trust. The privacy guarantee comes from code approval rather than from a mathematical bound, which means the protection is only as good as the reading somebody on your side gives the notebook before approving it, and an aggregate over a small overlap can identify individuals perfectly well while looking like an aggregate. Approving code you have not understood is the way this control fails, and it fails silently. We treat clean room notebook approval as a review with a named reviewer and a record of what they checked, in the same spirit as any other security review.

Marketplace, from the consumer side #

Marketplace is where third-party data products are discovered and acquired, with listings covering datasets, AI models, notebooks, apps, and Model Context Protocol servers. Underneath it is OpenSharing, which is why an acquired dataset arrives as a catalog rather than as an ingestion project. Some listings are available instantly once you accept the terms, others require provider approval or a transaction, and beyond the public marketplace a provider can run a private exchange whose listings are visible only to member consumers. Requesting access is done from a workspace, though working with the data afterwards is not restricted to one.

The consumer-side argument is about elapsed time rather than price. Acquiring a reference dataset conventionally means procurement, a delivery mechanism, a landing zone, an ingestion job, and a refresh that somebody now owns forever, whereas here it is a request and a catalog that shows up governed by Unity Catalog like anything else. Private exchanges are the underused half, because a company with many partners can run one rather than negotiating a delivery mechanism per partner.

The row we would draw attention to is the second one. Every platform can share well inside itself, and the difference is what happens when the other party is not on your platform, which describes most partners most of the time. An open protocol with clients for pandas, Spark, and the common business intelligence tools means the partner conversation is about the data rather than about which cloud they are willing to adopt.

Choosing between the three #

A partner needs to read your gold table on a continuing basis, so you create a share, add the tables, and grant it to a recipient, in the Databricks mode if they are on the platform and the open mode if they are not. Two companies want a joint answer neither can compute alone and neither is permitted to disclose the inputs for, so you use a clean room and accept that somebody must read the notebook properly. You need data your organisation does not have, so you look in Marketplace first and only build a pipeline when nothing there fits.

Our position is that the extract should be the exception rather than the default, and that the argument for making the change is not technical elegance. It is that a share can be revoked, an extract cannot, and every extract you have ever sent is still out there somewhere.

Table of Contents