Generative AI · Chapter 25

Context and Retrieval

Almost every disappointing answer we are asked to investigate turns out to be an information problem rather than an intelligence problem. This chapter is about the plumbing that decides what a model gets to read.

The context window is a budget, not a container #

The context window is the amount of text a model can consider in one call, and it is easy to read that number as a container waiting to be filled. It behaves much more like a budget, because several things are spending it before your retrieved documents arrive. Every call carries the system instructions, the definitions of any tools the model may call, the retrieved documents, whatever conversation history you are replaying, and the space the answer itself needs. A team that adds twelve tool definitions and a two thousand word style guide has spent a meaningful share of the window before the user has typed anything.

Filling what remains is not free either. Quality degrades rather than improves once the relevant passage is buried among fifteen irrelevant ones, because every additional near-miss is a chance to attend to the wrong thing. We see this most often where the retrieval parameter was set to twenty results on the reasoning that more context cannot hurt, and lowering it to five improves the evaluation score. Cost scales with the window on every call, so an extra three thousand tokens of boilerplate is a tax paid per request, for the life of the feature, at a volume nobody forecasts correctly.

Vector search on Databricks #

Semantic search converts text into a numeric vector whose position encodes meaning, so a question about missed payments finds a passage about arrears without the two sharing a word. Databricks provides this through AI Search, previously named Vector Search, and the part that matters architecturally is where the index comes from.

An index is created from a Unity Catalog table and kept synchronised with it. The source table is the system of record, the index is derived, and the platform keeps the second in step with the first. That synchronisation is the whole value proposition, because a hand-maintained index drifts the moment somebody edits a document without rerunning the loader, and stale retrieval is the most dangerous failure mode available. It produces neither an error nor an empty result, but a fluent, confident, correctly formatted answer based on a policy superseded in March, and nobody notices because the answer reads well.

CREATE OR REPLACE TABLE originations_prd.gold.policy_doc_chunks (
  chunk_id      STRING COMMENT 'Stable identifier, document id plus ordinal',
  document_id   STRING COMMENT 'Parent document, used for citation',
  section_path  STRING COMMENT 'Heading trail, for example Servicing > Arrears > Hardship',
  effective_from DATE,
  dealer_id     STRING COMMENT 'Owning tenant, null when the document is global',
  chunk_text    STRING
);

ALTER TABLE originations_prd.gold.policy_doc_chunks
  SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Change data feed lets the sync move only what changed rather than rebuilding, so enable it before creating the index. The index itself is one call.

from databricks.vector_search.client import VectorSearchClient

client = VectorSearchClient()

client.create_delta_sync_index(
    endpoint_name="originations-vector-search",
    index_name="originations_prd.gold.policy_doc_index",
    source_table_name="originations_prd.gold.policy_doc_chunks",
    primary_key="chunk_id",
    embedding_source_column="chunk_text",
    embedding_model_endpoint_name="databricks-gte-large-en",
    pipeline_type="TRIGGERED",
    columns_to_sync=[
        "chunk_id",
        "document_id",
        "section_path",
        "effective_from",
        "dealer_id",
        "chunk_text",
    ],
)

Because the embedding is generated from a column by a named endpoint, the index knows how its own vectors were produced. Triggered sync refreshes when you ask it to, which suits a corpus updated by a nightly pipeline, while continuous sync keeps the index close to live at a higher running cost. Choose on how much staleness the workload tolerates rather than on which sounds better.

Chunking is where most quality is won or lost #

A chunk is the unit that gets retrieved, so it is also the unit the model has to understand. Three rules cover most of it.

Cut on semantic boundaries rather than a fixed character count. Splitting every eight hundred characters will eventually halve a table, separate a heading from the paragraph it governs, and end a chunk mid-sentence. Sections, headings and paragraph boundaries are already present in most documents, and using them costs a little parsing work and returns a great deal of coherence.

Include enough surrounding context that a chunk is interpretable alone. A passage reading "this does not apply where the applicant is a sole trader" is useless without knowing what "this" is, so prepend the document title and the heading trail, and overlap slightly at boundaries.

Carry metadata on every chunk. Document identifier, section path, effective date, source system and owning tenant all belong as columns, because they are what lets you filter before searching, cite afterwards, and exclude a superseded revision without deleting it.

Retrieval is not only vectors #

The word retrieval has become a synonym for semantic search, and treating the two as the same thing is the second most common design mistake here.

Structured retrieval means letting the agent run a governed SQL query against a narrow allow-list of views. For anything countable that is not an alternative to semantic search but a straightforwardly more correct answer, because a question about how many applications were declined last month has an exact answer a warehouse can compute, while semantic search returns three paragraphs that mention declines. We check every corpus for questions of this shape early, since a document agent that cannot count is usually being asked to count.

Hybrid retrieval runs lexical keyword search alongside vector search and merges the candidate sets. Pure vector similarity is weak on exact identifiers, product codes, unusual proper nouns and negations, all of which keyword search handles precisely, so the union is more reliable than either alone.

Reranking retrieves a wide candidate set, perhaps fifty passages, then scores each against the question with a cross-encoder and keeps the best five. It is the most reliable single quality improvement available to most systems, and it costs latency, so it belongs behind a measurement rather than a preference.

Governed retrieval #

This is the part we will not compromise on. Filter at retrieval time using the identity of the person asking, and never retrieve broadly as a service principal on the understanding that the model will decline to mention what it should not have seen. If the retrieval can see it, treat it as disclosed, because a model instructed to withhold information it has already been given is a request rather than a control, and it fails to a prompt-injected document, a cleverly worded follow-up, or nothing more exotic than the model summarising its sources.

The mechanics are those covered in Multi-Tenant and Per-User Reporting and Unity Catalog, applied one layer up. Entitlements resolve from the caller before the search runs, and become a filter on the index rather than a post-processing step.

def retrieve_policy(question: str, caller: Caller, k: int = 5) -> list[Chunk]:
    """Retrieve policy passages the caller is entitled to read.

    Entitlements are resolved from the caller's identity and applied as an
    index filter, so unentitled chunks are never returned to the process.
    """
    scopes = entitlements.dealer_scopes_for(caller.user_name)
    if not scopes:
        return []

    response = index.similarity_search(
        query_text=question,
        columns=["chunk_id", "document_id", "section_path", "chunk_text"],
        filters={
            "dealer_id": scopes + [None],   # tenant documents plus global ones
            "effective_from <=": date.today().isoformat(),
        },
        num_results=k,
    )
    return [Chunk.from_row(row) for row in response["result"]["data_array"]]

Two details there are deliberate. An empty scope set returns nothing rather than falling through to an unfiltered search, because a filter that fails open is worse than no filter at all. The effective date predicate is applied at retrieval rather than left to the model, since a superseded policy in the context window is a superseded policy in the answer.

All four will index your documents competently, and OpenSearch and Azure AI Search are strong retrieval engines we are happy to work with. The difference is that on the other three the index is a separate system with its own copy of the data and its own access model, so the claim that a user only ever retrieves what they may read is something your team builds, tests and defends. On Databricks the index derives from a governed table and the entitlement lives in the same catalog as the data, which makes that claim a configuration rather than a project.

Advanced: the parts teams reach later #

Compaction and summarising history. A long conversation eventually exceeds the budget, and the usual answer is to summarise older turns into a compact note. Specifics are what that costs, because numbers, identifiers, exact quotes and the precise wording of a constraint are the first things summarisation discards, so a compacted conversation keeps the gist of what the customer wanted and loses the account number they gave you eleven turns ago. Keep structured facts in a slot that is never summarised, and compact only the prose.

Caching. Prompt caching stores the processed form of a prefix so that repeated calls sharing it skip most of the work, cutting both cost and time to first token on workloads with a large fixed preamble. Prefix order therefore matters, so put the stable material first, meaning system instructions, tool definitions and fixed reference text. Anything that changes invalidates the cache from that point onwards, which is why interleaving a timestamp or the user's name into the system prompt quietly costs more than it looks.

Metadata as retrieval context. Table and column descriptions are not documentation, they are retrieval inputs, and a well described schema is what lets an agent choose the right table. This is the argument the semantics pillar makes in The Five Pillars of Modern Analytics. Unity Catalog Business Semantics gives metric views a governed home, and the Genie knowledge store holds curated synonyms, sampled values and SQL instructions for a subject area. Both are context, both are versioned, and both improve answers without touching the model.

Evaluate retrieval separately from generation. When a system answers wrongly you need to know whether the passages were wrong or the reasoning was, and an end-to-end score cannot tell you. Build a labelled set naming, for each question, which chunk identifiers contain the answer, then measure recall at k, meaning the proportion of questions where a correct chunk appears in the top k results. If recall at five is sixty per cent, no amount of prompt engineering will fix the system.

Freshness and the window in between. Between a source table changing and the index reflecting it there is a gap, measured in minutes for continuous sync and in however often you trigger it otherwise. For a policy corpus that gap is harmless. For pricing, inventory, entitlements or anything a customer will act on it is a correctness problem, and the answer is usually to route those questions to structured retrieval against the live table rather than to make the index faster.

Long documents and multimodal content. Some material resists chunking, including contracts where a clause depends on definitions forty pages earlier and scanned documents where the meaning sits in a diagram. Here a larger context window genuinely is the right answer rather than a way of avoiding retrieval work, and the pattern that works is two-stage, where retrieval identifies the relevant document and the whole document then goes to a long-context model.

What we would tell a team starting today #

Write the labelled retrieval set before building the pipeline, because it converts every later argument about chunk size into a measurement. Chunk on structure, carry metadata, and resist raising the result count when quality disappoints, since the fix is almost always a better candidate set rather than a bigger one. Put the entitlement filter in on the first day, because retrofitting it means auditing every call site. What to do with a well-grounded model is Agents on Databricks, and what happens when one is not enough is Orchestrating Agents.

Table of Contents