Generative AI · Chapter 27

Orchestrating Agents

This chapter is about what happens when one agent is not enough, and about the engineering discipline that decides whether a multi-step AI system survives contact with production.

Tools before agents #

A tool is a function the model may call, described well enough that the model can work out when calling it is appropriate. Everything interesting an agent does, it does through tools, so the tool layer sets the ceiling for everything above it.

On Databricks, Unity Catalog functions can be registered as agent tools, and this is the single most important structural point in the chapter. A tool defined as a catalog function is a governed object with an owner, with EXECUTE grants deciding who may run it, with invocations in the audit log, and with its relationship to the tables underneath it visible in lineage. The ordinary alternative is a bare HTTP call from the agent's process to an internal service using a shared secret, invisible to the catalog and discoverable only by reading the agent's source.

CREATE OR REPLACE FUNCTION originations_prd.tools.loan_status(
  loan_number STRING COMMENT 'Loan number exactly as printed on the customer statement'
)
RETURNS TABLE (
  loan_number    STRING,
  status         STRING,
  days_past_due  INT,
  next_due_date  DATE,
  updated_at     TIMESTAMP
)
COMMENT 'Current servicing status for a single loan. Use when the customer names a
         specific loan number. Returns no rows when the loan is not visible to the
         caller. Does not return payment history; use payment_history for that.'
RETURN
  SELECT
    loan_number,
    status,
    days_past_due,
    next_due_date,
    updated_at
  FROM originations_prd.gold.loan_current_state
  WHERE loan_number = loan_status.loan_number;

GRANT EXECUTE ON FUNCTION originations_prd.tools.loan_status
  TO `servicing-agent-callers`;

The comments are not decoration. They are what the model reads when deciding whether to call this tool, so they belong in the register of a good docstring, saying what the tool returns, when to use it, and explicitly what it does not do. We have improved more agents by rewriting tool descriptions than by changing models. The underlying gold table carries the row filters from Multi-Tenant and Per-User Reporting, so the tool cannot return a loan the caller was not entitled to see, whatever the model was persuaded to ask for.

When one agent is enough #

One agent is enough more often than people expect, because a single agent with five well-described tools, a clear system prompt and good retrieval handles a surprising share of what teams reach for multi-agent architectures to solve, and it does so with one prompt to maintain, one trace to read and one bill to explain.

The genuine reasons to move beyond one agent are narrow. The tool list has grown past the point where the model reliably picks the right one, usually north of fifteen or twenty tools. Different parts of the problem need instructions that contradict each other when combined into one prompt. Or the steps have different risk profiles and you want a hard boundary between them. Those are real, and wanting the architecture diagram to look sophisticated is not.

The patterns that work #

Four arrangements cover almost everything we build, and each has an honest cost.

A router classifies the request and dispatches it to one specialised handler. It adds one cheap model call of latency, barely moves the token bill because the classifier can be a small model with a short prompt, and fails by misclassifying, which sends a question to a handler that answers it badly rather than declining. Give the router an explicit "none of these" branch and monitor how often it fires, because a rising unknown rate is the earliest signal that users want something the system was not built for.

A supervisor with specialised workers has one agent decompose a request, delegate to workers, and assemble their outputs. It is the most flexible pattern and the most expensive, because every worker call carries its own instructions and context, so a supervisor consulting four workers has roughly five times the token cost and, unless the workers run concurrently, five times the latency. It fails by looping, by delegating vaguely and getting vague answers back, and by summarising a worker's careful output into something wrong. Cap delegation depth and worker calls as hard limits in code rather than as instructions in a prompt.

A sequential pipeline gives each step a narrow job and passes the output of one to the next, which is much the easiest to reason about, test and price, because each step has a known prompt and a predictable size. It fails by propagating an error forwards, since step four cannot tell that step two invented a figure. Validate between steps, and prefer typed structured output over prose so validation is a schema check rather than a hope.

An evaluator-optimiser loop has one model produce a draft and another critique it against explicit criteria, iterating until the critique passes or a limit is reached. It genuinely improves writing, extraction and code generation, and it multiplies cost and latency by the number of rounds, with two rounds usually where the returns stop. Always set a hard iteration cap, because a loop whose exit condition is a model's opinion is a loop that can decline to exit.

Agent Bricks #

Agent Bricks is Databricks' product for building agents grounded in enterprise data. Rather than asking a team to hand-tune prompts and guess at configuration, it optimises quality and cost using synthetic data generation, custom evaluation and automated tuning, which is the work a careful team would otherwise do by hand over several weeks.

We want to describe this accurately rather than enthusiastically. It does not remove the need for an evaluation set that reflects your business, because the questions that matter still have to come from the people who will ask them, and it does not remove the governance boundary described below. What it removes is a meaningful amount of the mechanical iteration between a working prototype and something whose quality and unit cost you are willing to defend, which is the part of agent work most likely to be done badly under deadline pressure. Agent Bricks also carries a first-party version of the supervisor pattern, named Supervisor Agent, which coordinates Genie agents, agent endpoints, Unity Catalog functions, MCP servers and custom agents as subagents, so a team can assemble one without writing the delegation logic itself.

Determinism where it matters #

Governed Actions establishes the house position that agents propose and never act. The orchestration equivalent extends it. Wherever the sequence of steps is known, that sequence belongs in ordinary code calling models, rather than being described to a model that then decides what happens next.

The reasoning is not that models are bad at planning. It is that a known sequence expressed as code is testable, reviewable in a pull request, identical on every run, cheap because it spends no tokens deciding what to do, and debuggable with a stack trace. The same sequence expressed as a prompt is none of those things, and it will occasionally take a different route on a Tuesday for reasons nobody can reconstruct.

def handle(request: Request, caller: Caller) -> Reply:
    """Code decides the sequence. The model does the parts needing judgement."""
    intent = classify_intent(request.text)          # small model, constrained output

    if intent == "LOAN_STATUS":
        loan_number = extract_loan_number(request.text)
        if loan_number is None:
            return Reply.clarify("Which loan number are you asking about?")
        rows = tools.loan_status(loan_number, on_behalf_of=caller)
        if not rows:
            return Reply.not_found(loan_number)
        return summarise_status(request.text, rows)

    if intent == "POLICY_QUESTION":
        chunks = retrieve_policy(request.text, caller=caller, k=5)
        if not chunks:
            return Reply.refuse("I could not find a policy covering that.")
        return answer_with_citations(request.text, chunks)

    if intent == "HARDSHIP_REQUEST":
        proposal = draft_hardship_proposal(request.text, caller=caller)
        return Reply.needs_review(proposal)          # never executed here

    return Reply.refuse("I can help with loan status and servicing policy.")

There are three model calls in that function and none chooses what happens next. Classification, summarising retrieved rows and drafting a proposal are all judgement, while the control flow is a sequence of if statements a reviewer can read. Model-driven control flow remains right for genuinely open-ended problems, meaning research, exploratory investigation and anything where the number of steps depends on what earlier steps found. It is the wrong answer for a support assistant whose four intents were known when the ticket was written.

Durable execution #

A multi-step agent that dies halfway through is the worst outcome available, worse than failing at the start, because the first three steps may already have changed something and nothing knows how far it got. Deployments, container restarts, timeouts and rate limits all cause it, and the more steps a system has the more often it happens.

The answer is a durable execution engine, meaning one that persists workflow state so a run resumes at the step it reached rather than beginning again or failing silently. We use Temporal, and Fabric Harness is the client-side half of it, an open source TypeScript framework published under Apache 2.0 that provides durable execution and capability-level policy governance for the agent code sitting outside the platform, which for most of our clients is where the orchestration lives.

Advanced: running this at scale #

Token and latency budgets per request. Set both as numbers before building rather than after the invoice. A supervisor multiplies both by the number of workers it consults, and one calling workers that are themselves agents multiplies again, which is how a system that felt instant in testing takes eleven seconds with three concurrent users. Record tokens in, tokens out and duration per step, and alert per step rather than on the total, because the total says something is wrong while the per-step figure says which step.

Failure handling. Every tool needs its own timeout, since one shared timeout means the fastest tool waits as long as the slowest. Retries must be idempotent, so a tool that changes something needs an idempotency key and a retry after a timeout does not do the work twice. The case teams forget entirely is a tool returning something the model did not expect, whether an empty result, a validation error or a partial answer. Handle those in code and give the model a structured statement of what happened, because a raw stack trace pasted into the context window produces an apologetic hallucination rather than a useful reply.

Tracing a multi-step run. When a five-step system answers wrongly, the only question worth asking is which step went wrong, and without a trace that takes a day. MLflow 3 tracing gives you the run as a tree, and that tree maps onto the patterns above more directly than it first appears. A supervisor run is the tree. The supervisor's own reasoning is the span at the root, and every worker it consults hangs beneath as a child span carrying its own inputs, outputs, duration and status. Spans have declared types, so the retrieval inside a worker appears as a RETRIEVER span, the catalog function it calls appears as a TOOL span, and the model call that assembles the final answer appears as a CHAT_MODEL span, which means the tree reads as the architecture rather than as a wall of log lines. This is also what turns the multiplied cost from arithmetic into something visible. The budget note above says a supervisor consulting four workers costs roughly five times a single agent, and the trace is where you see which of the five actually spent it, that one worker was called twice because the supervisor did not believe the first answer, and that six seconds of an eight second response came from a worker that had to finish before the others could start. Evaluation attaches to those same traces, so the scorers you run in development are the scorers you can run over production traffic. Trace from the first prototype, because the run you most need to inspect is one that has already happened, and the full discipline sits in MLflow and the Model Lifecycle.

Evaluating end to end and per step. You need both, for different reasons. End-to-end evaluation is the only thing measuring what users experience, and it cannot tell you where a regression came from. Per-step evaluation localises the problem and can pass on every step while the assembled system still answers badly, usually because a step succeeded at its narrow job on an input that was already wrong. Run end-to-end as the gate and per-step as the diagnosis.

Human in the loop as a state, not an error path. Most systems bolt approval on as an exception, which is why it works badly. Model it as an ordinary state the workflow can occupy, with a persisted proposal, a defined timeout, an escalation route and an audit record of who approved it and when. A durable engine makes this natural, since a workflow waiting three days for a reviewer is the same construct as one waiting three seconds for an API. Once approval is a state rather than an exception, widening or narrowing what needs review becomes a policy change instead of a rewrite.

Step Functions deserves a specific compliment, because as a durable orchestrator it is mature, well-instrumented and something we happily build on, Semantic Kernel is a capable framework, and Bedrock Agents will get a competent agent running quickly. The distinction we draw is narrower than platform preference. On the other three a tool is a function in a compute service whose relationship to the governed data underneath it is something your team documents, whereas a Unity Catalog function shares a grant model and a lineage graph with the data it reads. That matters exactly when somebody asks which agents can reach which tables.

Where this leaves the boundary #

Everything in this chapter concerns systems that read, reason and propose. The moment an orchestrated run may change something in a system of record, the pattern discussion stops being an engineering preference and becomes a governance question, answered by the layer in Governed Actions. Orchestration decides how many model calls it takes to reach a recommendation, and the action layer decides whether that recommendation is permitted to happen.

Table of Contents