Storing · Chapter 07
Accounts and Workspaces
Almost every structural mistake we are asked to unwind was made in the first week, when somebody clicked through the workspace creation flow to unblock a demo and the demo became production. This chapter is about the decisions that are cheap on day one and expensive on day four hundred.
The short version
A Databricks account is the container your organisation buys once. Inside it you create workspaces, which are the rooms where people work and where computation happens. The important thing to know is that the valuable, permanent assets belong to the account rather than to any single room, meaning who your people are, what data exists, who may see it, and what everything costs. Workspaces are comparatively disposable. Teams who get this backwards end up with five copies of their user list and five versions of the truth about who can read customer data, and reconciling that later is a project rather than a task. Getting the shape right at the start costs an afternoon, and it is the highest-leverage afternoon in a Databricks adoption.
The account is the boundary that matters #
A Databricks account sits above workspaces and owns what should be shared, meaning identity, the metastore that holds your catalogs, billing, and the audit record. Workspaces are where people log in, where notebooks live, and where compute runs. The mistake teams make is treating a workspace as the unit of isolation for everything, which duplicates identity and catalogs per workspace and lets them drift apart.
It also helps to know that a workspace is not one thing. The control plane, holding the web application, job scheduling, and query coordination, runs in Databricks' own cloud account, while the compute plane, where clusters and warehouses actually process data, runs against your storage and, for classic compute, inside a network you own. Most security reviews want that split established early.
How many workspaces #
There is no universal answer, but there is a reliable starting shape. We separate workspaces where the blast radius genuinely differs, and we resist separating along boundaries that are really just team names, because teams, projects, and data domains are catalog and schema boundaries that Unity Catalog handles well.
| Boundary | Separate workspace | Rationale |
|---|---|---|
| Dev / staging / prod | Yes | Different blast radius, different access, different change control |
| Region or data residency | Yes | Often a hard legal requirement |
| Business unit with own budget | Usually | Cost attribution and independent admin |
| Team | No | Use catalogs and groups |
| Project or data product | No | Use schemas |
| Sandbox for experimentation | Sometimes | Only if you need genuinely permissive settings |
One constraint shapes this more than people expect. A metastore is regional, and a workspace attaches to the metastore in its own region. If you operate in three regions you have three metastores, and sharing data between them is a deliberate act using OpenSharing rather than something that happens for free. Plan the regional map before the workspace map.
Naming, and why we are strict about it #
Names are the cheapest form of documentation and the only one that appears in every error message, cost report, and audit log, so we settle on a convention before the first object exists. The pattern we use is {org}-{environment}-{purpose} for workspaces and {domain}_{environment} for catalogs, lowercase throughout, with environments spelled out as dev, stg, and prd rather than abbreviated inconsistently.
Infrastructure as code from the beginning #
Workspaces, catalogs, groups, grants, warehouses, and job definitions are all declarable. We use Terraform for account and workspace topology and Databricks Asset Bundles for the things that ship with application code, which is covered in CI/CD and Environments. The reason to do this on day one is not elegance; it is that a click-configured production workspace cannot be reproduced, which means it cannot be tested, which means every change to it is a change to production made by hand.
Beyond the basics #
Most teams stop after workspaces, groups, and tags. The controls below turn a working account into a defensible one, and they are considerably less well known.
Catalog isolation through workspace bindings. By default a catalog in the metastore is visible from every workspace attached to it, which quietly undermines the environment separation workspaces were meant to give you. Binding a catalog to specific workspaces, and marking it read-only in some of them, means a developer in the dev workspace cannot reach production data even when a grant is wrong. The binding is the belt and the grant is the braces.
Network connectivity configurations and serverless egress. Serverless compute does not run in your VPC, which raises the question of how you control where it can reach. A network connectivity configuration attaches to workspaces and gives you stable outbound identities and private endpoints from serverless compute to your storage and databases, plus egress policies restricting which destinations serverless workloads may contact at all. This answers the security review question that used to end serverless evaluations early.
Front-end private connectivity keeps user and API traffic off the public internet, and back-end private connectivity does the same for compute reaching the control plane. Combined with IP access lists, this gets you a workspace unreachable from anywhere except your network.
Customer-managed keys. You can supply your own keys for managed services, meaning notebooks and secrets in the control plane, and for workspace storage. The significant part is not the encryption, which exists either way, but that revoking the key revokes access.
Compliance security profiles. Where a workload carries a specific regime such as HIPAA, PCI-DSS, or FedRAMP, a workspace can be created with an enhanced profile enforcing hardened images, automatic cluster updates, and additional monitoring. This is a creation-time decision on most clouds, which is another reason to decide the workspace map before building it.
Budget policies and serverless attribution. Serverless compute has no cluster to tag, so attribution works differently. A budget policy carries the tags and is applied to the workloads that run under it. Teams who tagged everything carefully and then adopted serverless are often surprised by a growing pool of unattributed spend, and this is the mechanism that closes it.
System tables as the source of truth. Billing, audit, lineage, query history, and node timelines are exposed as governed tables in the system catalog, which makes cost and access questions answerable with SQL rather than with a support ticket. They are the foundation for the dashboards described in Observability.
Where account and workspace topology is decided on each platform
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Identity for the data platform | Account-level groups federated from your IdP, used by every workspace | IAM roles per account, plus Lake Formation principals | Entra ID mapped separately per Synapse workspace and service | Google groups plus per-project IAM |
| Catalog scope | One metastore per region, shared by every workspace in it | Glue Data Catalog per account and region | Purview for discovery, permissions live in each engine | Dataplex plus BigQuery datasets per project |
| Environment isolation | Separate workspaces plus catalog bindings | Separate AWS accounts wired with Organizations | Separate subscriptions or resource groups | Separate projects |
| Cost attribution | Compute tags and budget policies, queryable in system tables | Cost allocation tags across many services | Azure tags and cost management | Labels and billing export |
| Audit of data access | System tables in the same SQL surface as the data | CloudTrail plus Lake Formation logs, joined by you | Activity logs plus per-service diagnostics | Cloud Audit Logs plus BigQuery views |
Everything in that table is achievable on any of the three clouds. The difference is how many distinct services you assemble and keep in agreement to get there, and whether the resulting logs land somewhere you can query them alongside the data they describe.
Cost attribution starts here #
Every workspace, cluster policy, warehouse, job, and budget policy should carry tags that map onto how finance thinks about the business, typically cost centre, environment, and either product or client. Tags applied later do not retroactively label usage that already happened, so a month of untagged spend is a month you cannot explain. We enforce tags through cluster policies so untagged compute cannot start. Reading the resulting bill is covered in What It Costs.