Reference · Chapter 37
Glossary
Terms as this manual uses them. Where a Databricks product has been renamed, the older name is noted so that search still finds it.
The machinery #
Notebook: A document of runnable cells mixing code, results and prose. Where Python and exploratory work happen. A real file with versions and permissions, not scratch space.
Apache Spark: The open source engine under the analytical side, created by the founders of Databricks. It spreads one query's work across many machines. See The Vocabulary.
Cluster: A group of machines running Spark for you, one driver coordinating many workers.
SQL warehouse: Cluster capacity packaged for queries, sized in t-shirt sizes such as 2X-Small.
Serverless: Compute Databricks keeps warm and allocates in seconds, so there is nothing to size, start or switch off.
DBU: Databricks Unit, the billing unit, consumed per second while compute runs at a rate that depends on the compute type. See What It Actually Costs.
Parquet: The columnar file format Delta and Iceberg tables store their data in.
Platform concepts #
Account: The top-level Databricks entity that owns identity, the metastore, and billing. Workspaces belong to an account. See Accounts and Workspaces.
Workspace: The environment where people, notebooks, jobs, and compute live. Isolation boundary for environments and regions, not for teams.
Metastore: The account-level container holding catalogs. One per region, generally.
Unity Catalog: The governance layer covering data access, lineage, and audit across the account. See Unity Catalog.
Catalog: Top level of the three-part catalog.schema.object namespace. We use it as the domain and environment boundary.
Schema: Middle level of the namespace, previously called a database. We use it as the medallion layer or data product boundary.
External location: A registered storage path plus the credential permitting access to it, which is how Unity Catalog governs data it does not manage directly.
Delta Sharing: Mechanism for granting read access to a table across organisational boundaries without copying data.
Data and storage #
Delta Lake: The default table format: Parquet files plus a transaction log providing atomicity, time travel, and concurrent reads.
Iceberg: An alternative open table format that Databricks reads and writes, and can expose through a REST catalog to external engines. See Tables and Storage.
Managed table: A table whose storage location and lifecycle Databricks owns. Our default.
External table: A table whose files live in a location you manage. Used when another system must write the same files.
Liquid clustering: Data layout mechanism that supersedes Hive-style partitioning for most tables, adapting to query patterns without directory-level partitioning.
Predictive optimisation: Automatic compaction, statistics, and clustering maintenance driven by observed query patterns rather than by a schedule.
Time travel: Querying or restoring a Delta table as of a prior version or timestamp, bounded by the table's retention settings.
Pipelines and orchestration #
Auto Loader: Incremental file ingestion from object storage with schema inference and evolution, without you tracking which files were processed. See Ingestion.
Lakeflow Declarative Pipelines: The declarative pipeline framework, previously called Delta Live Tables or DLT. See Pipelines.
Expectation: A declared data quality rule on a pipeline table, enforced by warning, dropping the row, or failing the pipeline.
Streaming table: A pipeline table that processes each input record once, appropriate for ingestion and append-heavy sources.
Materialised view: A pipeline table that recomputes a result, incrementally where the query shape allows, appropriate for aggregations and gold-facing tables.
Job: A scheduled or triggered multi-task graph. The platform's orchestration primitive. See Orchestration.
Asset Bundle: Declarative definition of jobs, pipelines, and resources with per-environment overrides, used for deployment. See CI/CD and Environments.
Medallion layers #
Bronze: Raw, as received, append-only, with ingestion metadata. Promises fidelity.
Silver: Typed, deduplicated, validated, conformed. Promises correctness.
Gold: Shaped for a named consumer. Promises fitness for purpose.
Quarantine: Where records failing validation go, with the reason attached, so that losses stay visible. See The Medallion Architecture.
AI and agents #
Model Serving: Managed endpoints hosting foundation models and your own, with logging into inference tables.
AI Search: Previously Vector Search. Managed index over text in Unity Catalog tables, kept synchronised with the source.
MLflow: Experiment tracking, model and prompt versioning, and evaluation.
Lakebase: The Postgres-compatible operational layer alongside the lakehouse, for transactional and low-latency workloads. See Lakebase and Operational Data.
Groundedness: Whether a generated answer is supported by the context actually retrieved, as distinct from whether it happens to be true.
TechFabric terms #
Governed action: A proposed effect on a system of record, evaluated against policy before execution, producing pass, warn, or block with evidence. See Governed Actions.
Decision evidence: The record emitted by a policy evaluation, containing the decision identifier, the policy bundle revision, and a hash of the inputs.
Fabric Airlift: TechFabric's Databricks migration accelerator.
Fabric Harness: TechFabric's open source TypeScript framework for governed data and AI agents, built on durable execution.
Platform mechanics #
Branch (Lakebase): A full copy of a database created by pointer rather than by copying bytes, so it takes about a second and costs almost nothing until something diverges. See Lakebase.
Copy-on-write storage: Storage where a copy shares its pages with the original until one of them changes, which is what makes branching cheap.
Separated storage and compute: An architecture where the database engine is an ephemeral thing pointed at durable versioned storage, rather than a machine with a disk attached.
Lakeflow Connect: The managed source connectors for pulling from SaaS applications and databases without writing ingestion code.
Lakehouse Federation: Querying an external system in place through Unity Catalog, without ingesting it first.
Clean room: A governed environment where two organisations compute over combined data without either seeing the other's rows.
Deletion vector: A marker recording that a row is deleted without rewriting the file that holds it, which makes deletes and updates much cheaper.
Shallow clone: A copy of a table that shares the underlying files with its source until written to, useful for testing against production-shaped data.
Photon: The vectorised query engine that speeds up most SQL and DataFrame work, billed at a higher rate and usually cheaper overall because the work finishes sooner.
Shapes of work #
OLTP: Online transaction processing. Reading or writing a few rows by key, fast, with strict correctness. What an application does. See The Vocabulary.
OLAP: Online analytical processing. Scanning many rows to compute an aggregate. What reporting does.
HTAP: Hybrid transactional/analytical processing. The previous generation's attempt to unify the two inside one engine.
Zero ETL: The traditional clouds' term for vendor-operated replication between an operational and an analytical store. The pipeline still exists; the vendor runs it.
LTAP: Lake transactional/analytical processing. Databricks' term for unifying the two at the storage layer, one copy on open storage with independent compute. See LTAP.
Analytics and reporting #
Unity Catalog Business Semantics: The platform's semantic layer, made of metric views and agent metadata.
Metric view: A SQL-defined business measure and its dimensions, living in Unity Catalog rather than inside a BI tool, readable by any engine that speaks SQL.
Agent metadata: Contextual knowledge an AI agent uses to interpret a question against the semantic layer.
AI/BI Dashboards: Dashboards that run in place on governed data and metric views, with no extract and no per-seat licence limit.
Genie: The conversational analytics agent, scoped per subject area and grounded in Unity Catalog and Business Semantics.
Knowledge store: The curated context for a Genie space: descriptions, synonyms, sampled values, value dictionaries and SQL instructions.
Row filter: A SQL function attached to a table that removes rows a principal may not see, evaluated per row at query time. See Multi-Tenant and Per-User Reporting.
Column mask: A SQL function attached to a column that transforms or passes through its value depending on who is asking.
Ingestion and change data #
Change data capture: Reading a source system's change log so that a target stays in sync without re-reading everything.
Ingestion Gateway: The Lakeflow Connect component that extracts snapshots, change logs and metadata from a source database.
Change feed: A source system's own ordered record of changes, such as the Azure Cosmos DB change feed. See Real-Time and Change Data Capture.
AUTO CDC: The declarative way to apply change records into a target table, with keys, a sequencing column and delete handling, supporting SCD type 1 and type 2.
Models #
Model alias: A moving pointer to a model version, such as champion or challenger, which is how promotion happens without copying artefacts.
Inference table: A governed table capturing the request and response traffic of a serving endpoint, which turns production traffic into an evaluation dataset.
Generative AI #
Foundation model: A large pretrained model served behind an endpoint, either hosted by Databricks or proxied from another provider. See Foundation Models and the AI Gateway.
AI Gateway: The governance layer in front of every model call, where authentication, rate limiting, logging and guardrails are applied once rather than per application.
Pay-per-token: Billing by input and output tokens against a shared endpoint. The right default until load is predictable.
Provisioned throughput: Reserved capacity on a serving endpoint, which trades a fixed cost for predictable latency under known load.
AI Functions: Calling a model from SQL, so classification or extraction happens inside a pipeline rather than in a separate service.
Vector search: A managed index over text in Unity Catalog tables, kept synchronised with the source table. See Context and Retrieval.
Chunk: A passage of a document indexed as one retrievable unit. Chunking on semantic boundaries rather than a character count is what makes retrieval interpretable.
Reranking: Narrowing a wide set of retrieval candidates to the few that actually go into the prompt.
Context window: The total budget of tokens a model can attend to in one call, shared between instructions, tool definitions, retrieved text, history and the answer.
Prompt caching: Reusing the computed form of a repeated prompt prefix to cut cost and latency.
Tool: A function a model may call. On Databricks a Unity Catalog function registered as a tool inherits the catalog's permissions and lineage. See Orchestrating Agents.
Router, supervisor, sequential pipeline: The three orchestration shapes that survive production, in ascending order of cost and latency.
Agent Bricks: The Databricks product for building agents grounded in enterprise data, which tunes quality and cost using synthetic data and custom evaluation.
Durable execution: Running a multi-step process so that a failure resumes where it stopped rather than restarting or half-completing.
Learning #
Free Edition: The no-cost Databricks account that replaced Community Edition. Serverless compute, one workspace, one metastore, and explicitly not for commercial use. See Getting Started Free.