Generative AI · Chapter 24

Foundation Models and the AI Gateway

Renting a language model is easy and every cloud will sell you one by the token. This chapter is about the harder question, which is how an organisation ends up with one governed way of calling models rather than forty ungoverned ones.

The four ways to get a model #

Databricks Model Serving deploys agents, chains, generative models and classical machine learning models on the same infrastructure, which means the four options below differ in how the model is obtained and paid for rather than in how your application calls it.

Pay-per-token foundation model endpoints are the default and the right starting point. Popular open and commercial models are available immediately with no capacity to provision, and you pay for the tokens you send and receive. Cost scales linearly with usage and falls to nothing when nobody is asking questions, which suits development, bursty internal tools and anything whose volume you cannot predict.

Provisioned throughput reserves capacity for a model, giving predictable latency under sustained load and a bill that reflects the reserved capacity rather than the traffic. It is right when a workload has a floor, when latency variance is a product problem rather than an annoyance, or when a batch job would otherwise be rate limited by a shared pool. It is wrong for anything intermittent, because reserved capacity that sits idle is the same mistake as a cluster left running overnight.

External models are third-party endpoints, meaning OpenAI, Anthropic and others, registered in Databricks and proxied through the gateway. You keep the vendor relationship and the vendor's billing, and you gain a single place where the credentials live and where the traffic is logged. This is the option most organisations underuse, because it lets a team keep the model it has already chosen while bringing the calls inside the governance boundary.

Fine-tuned and custom models are models you have trained or adapted and now host yourself, registered in Unity Catalog through MLflow and served from the same endpoint layer. The cost shape is the hosting cost plus the training that produced it, and the reason to choose it is a task where a smaller specialised model beats a larger general one on both quality and unit cost. Agent Bricks sits above this, building agents grounded in your enterprise data and optimising quality and cost using synthetic data, custom evaluation and automated tuning, which is a fair description of the work a team would otherwise do by hand.

The gateway is the argument #

The AI Gateway, also called the Unity AI Gateway, applies governance across the models and the MCP servers an organisation uses, and its features are configured on a serving endpoint rather than as one setting for the whole workspace. The documentation names six. Permission and rate limiting decides who has access and how much of it, expressed in queries per minute or tokens per minute. Payload logging writes requests and responses into inference tables managed by Unity Catalog. Usage tracking records operational usage and the cost attached to it into system tables, which is where system.serving.endpoint_usage comes from. AI guardrails screen what goes in and what comes back, covering unsafe content and personally identifiable information that you may either block or mask. Fallbacks route a request to the other models served by the same endpoint so that a failure degrades rather than stops. Traffic splitting sends a stated percentage of traffic to each model behind the endpoint. Not every feature applies to every kind of endpoint, and fallbacks in particular are configured on external model endpoints, so check what your endpoint type supports before you design around one of them. An application calling through the gateway does not need to know whether the model behind it is hosted by Databricks or by a third party.

The alternative is the one almost every organisation arrives at accidentally. A product team gets an API key, a second team gets its own, a contractor puts one in a serverless function, and eighteen months later nobody can answer three questions that are not unreasonable to ask. Which systems are sending customer data to a model vendor, what did they send, and what would happen to the monthly bill if one of them entered a retry loop. None of those answers can be reconstructed later, because the evidence was never collected.

Three properties are worth naming. The credential for an external vendor lives in the gateway rather than in each application's configuration, a rate limit can be lowered centrally during an incident without redeploying anything, and the record of what was sent and returned lands in a governed table while the token counts behind it land in a system table nobody had to instrument. Between them they turn a set of promises into a control.

AI Functions bring the model into the query #

The consequence of endpoints being a platform service is that SQL can call one. Classification, extraction, summarisation and translation stop being a separate service with its own queue and become an expression inside a pipeline, which removes an entire class of orchestration work.

CREATE OR REPLACE TABLE ops.support.tickets_classified AS
SELECT
  ticket_id,
  received_at,
  body,
  ai_query(
    'databricks-meta-llama-3-3-70b-instruct',
    'Classify this support ticket as BILLING, TECHNICAL, ACCOUNT or OTHER. ' ||
    'Reply with one word only. Ticket: ' || body
  ) AS category
FROM ops.support.tickets_raw
WHERE received_at >= current_date() - INTERVAL 1 DAY;

Two cautions come with that convenience. The first is that a model call per row is a cost per row, so a WHERE clause is a budget control and an unfiltered rebuild of a large table is an expensive accident. The second is that model output is not deterministic, so a column produced this way should be treated as a derived attribute with a recorded model version behind it rather than as a fact, which is why the silver and gold separation in Medallion Architecture matters more here rather than less.

Application code reaches the same endpoints through an OpenAI-compatible client, which means most existing code changes only its base URL and its model name.

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DATABRICKS_TOKEN"],
    base_url=f"{os.environ['DATABRICKS_HOST']}/serving-endpoints",
)

response = client.chat.completions.create(
    model="databricks-meta-llama-3-3-70b-instruct",
    messages=[
        {"role": "system", "content": "Summarise the ticket in one sentence. Use only what is given."},
        {"role": "user", "content": ticket_body},
    ],
    temperature=0,
    max_tokens=120,
)

summary = response.choices[0].message.content

Payload logging is what makes evaluation possible #

Enabling payload logging on an endpoint captures requests and responses into inference tables, which are ordinary Unity Catalog tables you can query, join and retain under the same policies as everything else. This is the least exciting feature in the chapter and the one whose absence hurts most, because evaluation of a system in production requires the production traffic, and traffic that was not captured cannot be recovered.

With those tables you can measure the real distribution of prompt lengths rather than the one you assumed, attribute spend to an endpoint and a caller, sample live traffic and score it with a judge model, and build an evaluation set out of questions users actually asked. MLflow and the Model Lifecycle covers the evaluation machinery, and Agents on Databricks covers the loop that feeds low-scoring production answers back into an offline set.

AWS and Azure both do parts of this well, and Bedrock Guardrails in particular is a mature safety layer that we recommend without hesitation to teams already committed to Bedrock. The distinction is scope rather than quality. Each of those guardrail products governs its own vendor's models, so an organisation running Bedrock for one product and Azure OpenAI for another has two control planes and no consolidated record, while the gateway's claim is that the doorway is the same regardless of who hosts the model behind it.

Advanced: the parts teams reach later #

Rate limits and fallbacks. Limits belong per endpoint and per user rather than as one global number, because the intent is to stop one runaway consumer rather than to throttle everybody. Fallbacks route a request to the other models served by the same endpoint when the first one fails, which converts an outage into slower or slightly different answers. Configure the fallback deliberately, because silently answering from a different model is acceptable for a summariser and unacceptable for anything whose output is compared over time.

Cost attribution per endpoint. Endpoint-level usage combined with the caller identity in the inference tables gives cost per team, per feature and per query. Do this before anyone asks for it, because the first budget conversation about generative spend arrives with the invoice and goes very differently when the answer is already in a table.

PII handling before the prompt leaves the boundary. Decide explicitly what may cross to a vendor. Redaction or tokenisation of identifiers before the call, applied in the pipeline or the gateway rather than in each application, is the difference between a data protection assessment that concludes quickly and one that stalls. Column masks in Unity Catalog are the natural place to define what is sensitive once, so the answer does not vary by whoever wrote the prompt.

Version pinning. A floating model alias that quietly moves to a newer version will change your outputs, your evaluation scores and sometimes your parsing, on a date you did not choose. Pin the version in production, record the pinned version in the inference record, and move it with the same deliberateness as any other deployment. Reproducibility is a compliance requirement in regulated work and a debugging necessity everywhere else, and neither survives a model that changed underneath the application.

MCP through the same gateway. Tool servers reached over the Model Context Protocol are the same governance problem in a different costume, because a tool call reaches systems of record rather than merely reading text. Routing MCP servers through the gateway means the same authentication, the same limits and the same log covers what the model reads and what it invokes, which is the precondition for the controls in Governed Actions.

What we would tell a team starting today #

Choose one model to begin with rather than spending a week comparing them, because the gap between the leading models is smaller than the gap between a good and a bad retrieval step, and switching later is a configuration change once the calls go through a gateway. Route everything through that gateway from the first prototype, since retrofitting it means finding every key issued in the meantime. Turn on payload logging before you have traffic worth logging, and pin your model versions before the first time an output changes without explanation. Grounding those calls in your own data and turning them into something that can act are Context and Retrieval and Agents on Databricks.

Table of Contents