Operating · Chapter 31

Security and Identity

Most of the security posture of a Databricks estate is decided by the data governance choices covered in Unity Catalog. This chapter covers the rest, and particularly the parts that are easy to leave until an audit forces the issue.

Identity comes from one place #

Users and groups federate from the corporate identity provider into the Databricks account, and provisioning is automated so that joining, moving, and leaving are reflected without a manual step. Groups are defined once at account level and assigned into workspaces, rather than being recreated per workspace, which is how estates end up with three groups of the same name holding different members.

The reason automated provisioning matters more than it appears to is offboarding. A manually managed group membership survives someone's departure, and the gap between an employee leaving and their access being removed is the single most common finding in the access reviews we take part in.

Service principals, not personal accounts, run automated work. A pipeline authenticating as the engineer who wrote it will break when that engineer changes role, and until then it grants the pipeline whatever that person can do, which is invariably more than the pipeline needs. Service principals authenticate with OAuth machine-to-machine credentials rather than long-lived personal access tokens, and we disable personal access tokens entirely where the tooling allows it.

The important detail in that diagram is the storage credential. Users do not hold cloud permissions on the underlying buckets; Unity Catalog holds them and vends short-lived credentials for the specific data a query is entitled to. That is what stops the common failure of an assembled stack, where table-level rules are enforced by the query engine and then bypassed entirely by anyone with direct object storage access.

Secrets #

Secrets belong in a secret scope backed by the cloud provider's key vault, referenced by name at runtime, and never in notebook source, job parameters, cluster environment variables, or a configuration file in the repository.

Rotation should be scheduled rather than event-driven, because a credential that is only rotated after an incident has by definition never been rotated. We prefer short-lived credentials and workload identity federation wherever the integration supports it, which removes the rotation question rather than answering it.

Network posture #

The defaults are usable and they are permissive. Three decisions are worth making deliberately: whether the workspace has a public endpoint at all, whether traffic to cloud storage stays on the provider's private network, and what egress from compute is permitted.

Private connectivity to the control plane and to storage is the control most estates should adopt, and it is largely a configuration exercise. IP access lists are a cheap additional layer for the workspace endpoint. Restricting egress is the control with the most operational friction, because it breaks package installation from public repositories, and the answer is an internal mirror rather than an exception. Teams that skip the mirror end up with a permanent exception, which is the same as not having the control.

Serverless compute deserves specific thought here, because there is no cluster in your network to attach rules to. The platform provides network policies for serverless workloads, controlling which destinations they may reach, and this needs configuring explicitly rather than assuming the serverless case inherits the classic one.

Data protection #

Encryption at rest and in transit is on by default and rarely needs attention beyond confirming that the key management arrangement satisfies the compliance requirement, which usually means customer-managed keys for regulated data.

The controls that need actual design work are classification and masking. We tag columns holding personal or regulated data in Unity Catalog, then drive access decisions from those tags rather than writing a policy per table.

Deletion deserves specific attention, because Delta's history means a deleted row is not immediately gone. Satisfying a deletion request requires understanding retention settings and the vacuum schedule, and it is worth working that procedure out before someone asks for it under a deadline.

Audit #

Audit logs cover authentication, data access, and administrative change. They are only useful if they leave the platform and land somewhere with independent retention and independent access, so configure delivery to the central logging estate on day one; audit logs configured after an incident do not cover the incident. The same events are queryable in the system tables, which makes routine questions such as who read this table last month a SQL query rather than a log-forwarding project.

The advanced controls worth knowing about #

Attribute-based access control lets you write a policy once against a column tag and have it apply to every table carrying that tag, including tables created next year. The alternative, a masking function granted per table, drifts the moment someone forgets, and the failure mode is silent because nothing errors when a policy is missing.

Row filters and column masks are enforced by the catalog rather than by a view someone might route around. Users query the real table and receive the rows and columns they are entitled to, which means one table definition rather than a proliferation of restricted views to keep in step.

Delta Sharing exposes data to external parties without copying it, revocable and auditable, which replaces the extract-and-email pattern that is the largest uncontrolled data flow in most organisations.

Clean rooms allow two parties to compute jointly on data neither can see, which is the compliant answer to a set of partner-analytics requests that otherwise turn into a data transfer nobody is comfortable with.

Compliance security profiles enable hardened baselines and enhanced monitoring for workloads under regimes such as HIPAA or PCI, including automatic cluster image updates so long-running compute does not sit on stale images.

System tables for access review turn the quarterly review from a screenshot exercise into a query, and lineage answers the question a grant list cannot, which is what this access actually reaches downstream.

Naming the controls an auditor asks for #

The controls above are easier to procure when they are called by the names they carry in a cloud bill, a change request, and an audit finding.

Private connectivity. The product is AWS PrivateLink on AWS, Azure Private Link on Azure, and Private Service Connect on Google Cloud, and on all three it buys two things that are bought and approved separately. Inbound connectivity, which the documentation also calls the front-end connection, secures the path from your users to the workspace web application and REST API. Classic compute plane connectivity, the back-end connection, secures the path from clusters running in your own network to the control plane, including the secure cluster connectivity relay that clusters use instead of accepting inbound connections. On AWS the back-end case needs two endpoints rather than one, a workspace endpoint and a relay endpoint, which catches teams who budgeted for a single line item. Serverless compute is a third question again, answered by the network connectivity configurations described in Accounts and Workspaces.

Customer-managed keys. There are two configurations and they cover different data in different places. A managed services key encrypts control plane data, meaning notebook source and metadata, secrets held through the secrets API, SQL queries and query history, and the artefacts behind dashboards and model serving. A workspace storage key encrypts storage in your own cloud account, meaning the workspace storage bucket that holds the DBFS root, job results, and query results, and optionally the volumes attached to classic compute nodes. Serverless workspaces take the managed services key, while classic workspaces can take both. Regulated industries ask for these not because the data would otherwise be unencrypted, since it is encrypted either way, but because key custody is a control they can demonstrate: the key sits in your own key management service, its use is logged there, and withdrawing it withdraws access.

Token hygiene. Personal access tokens are the thing to drive toward zero, and the documentation now describes them as a legacy authentication method. Service principals authenticating with OAuth machine-to-machine credentials replace them for everything automated. Where tokens still exist, three settings do most of the work: the maximum lifetime applied to newly created tokens, which defaults to a period measured in years and should be cut to under ninety days; the permissions governing who may create and use tokens at all; and the workspace switch that disables personal access token authentication outright, which is the end state we aim for. Existing tokens are listable and revocable through the token management API and the account console, so a quarterly review can enumerate them rather than assume.

Identity provisioning. Automatic identity management brings users and groups across from Microsoft Entra ID or Okta without a SCIM application to maintain, provisioning a user on first login, and on Entra ID it carries nested groups and service principals as well, which SCIM provisioning does not. It requires identity federation to be in place first. Where SCIM is already running it continues to manage the identities it created, so the two coexist through a transition rather than fighting over the same users.

Secrets as governed objects. Alongside the workspace secret scopes described earlier in this chapter, a secret can now be a Unity Catalog securable in its own right, named with the same three levels as a table and governed by the same privileges and the same audit trail. For new work we prefer that, because it removes one of the last things on the platform that carried its own separate permission model.

The reviews that catch real problems #

Two recurring reviews find more than any tooling we have used. The first is a quarterly access review listing every group, its members, and its grants, sent to the data owner rather than to the platform team, because the platform team cannot know that a person no longer needs access and the owner can. The second is a quarterly review of direct grants, service principals, and exceptions, with the expectation that the direct grant list is empty. It never is on the first run, and emptying it is where the interesting discoveries happen.

Table of Contents