Generative AI · Chapter 26
Agents on Databricks
This is the fastest moving chapter in the manual and the one where our opinions are held with the most awareness that they may need revising. What follows is what we currently build and why.
The short version
An agent is software that uses a language model to answer questions or carry out tasks, usually by looking things up in your own data first. The hard part is almost never the model, which you rent by the token from several vendors who are all roughly comparable. The hard part is making sure the agent looks at the right information, that it only sees what the person asking is allowed to see, and that you can prove afterwards what it was told and what it said. Building it next to the data, inside the permission and audit system that already governs that data, removes most of that difficulty rather than deferring it. The rest of the work is measurement, meaning somebody decides what a good answer looks like and writes down enough real questions to test it.
Why agents belong next to the data #
The case for building agents on Databricks rather than beside it is governance and lineage. An agent that retrieves from governed tables inherits the access model that already exists, its retrievals are audited, and the relationship between what it read and what it said is traceable. An agent built against a separate vector store fed by a nightly export inherits none of that, and its security review starts from scratch on infrastructure that cannot say who was allowed to see which chunk. That is the whole argument, and it is enough; where a client has a mature stack elsewhere we integrate rather than relocate.
The pieces #
Model serving hosts foundation models and your own behind an endpoint with authentication, rate limiting, and request logging. Inference tables capture the traffic into governed tables, which is what makes evaluation possible after the fact rather than only in a test harness.
Vector search indexes text from Unity Catalog tables and keeps the index synchronised with the source. The synchronisation is the valuable part; hand-maintained indexes drift, and a stale index produces confidently wrong retrieval that nobody notices because the answer still reads well.
MLflow tracks experiments, versions models and prompts, and holds evaluation runs. We register agents in Unity Catalog as models so they carry the same governance, lineage, and promotion path as anything else, which puts them on the route described in CI/CD.
Agent frameworks provide scaffolding for common patterns, and what matters is only that the framework produces an artefact that can be registered, versioned, and evaluated. Our own Fabric Harness, an Apache-2.0 TypeScript framework, exists because much of our client work is TypeScript and because we wanted durable execution underneath tool calls.
Evaluation is the whole job #
The set should come from the people who will use the agent rather than the people building it. Builders write questions the system can answer. Users write the question with an ambiguous premise, the one requiring data the agent cannot see, and the one that is really two questions in a single sentence.
Each run reports correctness against the known answer, groundedness in retrieved context, refusal rate on questions that should be refused, latency, and cost per query. Refusal is the metric teams skip and the one that most often reveals a problem, because an agent that never refuses will confidently answer a question about data it does not have.
Retrieval quality beats model quality #
Most agent failures we investigate are retrieval failures wearing a costume. The model was fine; it was handed the wrong three paragraphs.
The fixes are unglamorous. Chunk at semantic boundaries rather than a fixed character count, include enough surrounding context that each chunk is interpretable alone, put table and column descriptions to work because a well described schema is retrieval context, and filter by the user's actual permissions rather than retrieving broadly and hoping the model declines to mention what it saw.
Advanced: the things most teams have not reached yet #
The material above gets a useful agent into production. What follows is where the difference between a working agent and a durable one lives, and it is consistently the part clients did not know existed.
Hybrid retrieval and reranking. Pure vector similarity is poor at exact identifiers, product codes, and negations. Running lexical search alongside vector search, merging the candidate sets, then reranking with a cross-encoder before anything reaches the model is the largest quality improvement available to most systems. It costs latency, so it belongs behind a measurement rather than a preference.
Structured retrieval instead of text retrieval. A surprising share of questions asked of a document agent are aggregate questions about tables. A governed SQL tool, with a narrow allow-list of views rather than the whole catalog, answers those correctly instead of retrieving three paragraphs that mention revenue. The view layer is the security boundary, enforced by the warehouse rather than the prompt.
Online evaluation from inference tables. The offline set proves a change did not regress the questions you thought of; inference tables tell you about the ones you did not. We sample production traffic, score it with a judge model on groundedness and refusal, and route low scores into a review queue feeding new cases back into the offline set. That loop stops an evaluation set ageing into irrelevance.
Prompt and index versioning. A prompt change is a production change. Registering prompts alongside models, so that an inference record names the prompt revision, the index revision, and the model version behind it, is what makes an incident investigable months later.
Cost-aware routing. Not every question needs the largest model. Routing by classified difficulty, with a cheaper model handling the majority and escalating on low confidence, takes a substantial fraction off the bill with no measurable quality loss, provided the threshold comes from the evaluation set rather than instinct.
Building a governed retrieval agent on each platform
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Foundation and custom models | Model serving, one endpoint for both | Bedrock plus SageMaker, two services | Azure AI Foundry plus Azure ML | Vertex AI |
| Index synced from source tables | Managed, from Unity Catalog | Build it into OpenSearch or pgvector | Build it into AI Search | Build it into Vertex AI Vector Search |
| Retrieval under the table's own policies | One Unity Catalog model | Lake Formation governs tables, not the index | Purview labels data, index is separate | Dataplex governs tables, index is separate |
| Request logging into governed tables | Inference tables, queryable in SQL | CloudWatch and S3, wire it up | Log Analytics, wire it up | Cloud Logging, wire it up |
| Lineage from source table to answer | Inherited from the catalog | Assemble it | Assemble it | Assemble it |
Every one of these platforms can build the same agent, and the difference is how much governance is inherited rather than assembled. On the other three the index is a separate system with a separate access model, so the claim that the agent sees only what the user can see is something your team implements and defends.
Agents propose, systems act #
The distinction we hold most firmly is between an agent that produces a recommendation and one that changes something in the world. Reading, summarising, drafting, and recommending are one risk class; writing to a system of record, moving money, contacting a customer, or changing an account's status is a completely different one. Everything in this chapter concerns the first class. The second needs a governance layer outside the model, and it is the subject of Governed Actions.
Costs move differently here #
Agent workloads bill on tokens and endpoint uptime rather than the compute patterns the rest of the platform has taught your team to reason about, and cost per query varies with the input in a way a SQL query's does not. Instrument it from day one, set budget alerts on serving endpoints specifically, and remember that a retrieval change improving quality by adding context also multiplies the bill. Record tokens in and out alongside the evaluation score, so quality and spend are read from the same table.