Databricks Field Guide

Generative AI · Chapter 21

Governed actions for Databricks AI agents and write-back

Reviewed by TechFabric engineers. How we write this guide →

A governed action is a change an AI agent wants to make in a system of record, such as a refund, a CRM update or a write back to Lakebase, that's checked against versioned policy before it happens, recorded as evidence, and executed by a durable workflow rather than by the agent. Unity Catalog governs who may read data. Nothing in Databricks decides whether this refund, for this customer, at this amount, should happen automatically, so governed write-back is a layer you add.

The day an agent goes from answering questions to issuing a refund, the governance you built for reads stops covering you. This chapter is how we'd govern the write, with a proposal object, an Open Policy Agent check, a decision table in Lakebase and a Temporal workflow, each with code that runs as written. It's the most opinionated chapter in the guide.

The short version

Every major cloud platform is good at controlling who can read which data, and none of them controls what a system is then allowed to do about it. Those are different questions, and the second carries the business risk, because reading a customer's balance harms nobody while issuing a refund or cancelling a policy has consequences no data permission model inspects. Once software driven by a language model starts taking actions rather than producing text, the organisation needs a layer that checks each proposed action against written rules before it happens, records the decision for later audit, and guarantees the action either completes fully or not at all. Our rule is that the model proposes and never acts.

The gap #

Unity Catalog governs data access with real rigour. It knows who may read a table, which columns they may see, and which rows are visible to them, and it records every access. We'd rely on it completely for that, as Unity Catalog describes.

It says nothing about actions. When a system reads a customer's account, decides that a payoff quote should be regenerated, and calls an API that regenerates it, no data access policy has been violated. The read was permitted, and the write happened somewhere else entirely, in a system of record that Unity Catalog does not govern and cannot see.

For traditional analytics this gap doesn't matter, because analytics reads and doesn't act. For agentic systems it's the entire risk surface.

Databricks does govern the edges of it. A tool registered as a Unity Catalog function needs EXECUTE, and its calls land in the audit log. Unity Gateway governs which identities may reach which models, MCP servers and tools. An app running on behalf of the user writes with that user's grants rather than a service principal's. Each of those answers "may this identity call this thing". None answers "should this refund, for this customer, at this amount, happen automatically", and that's the question an auditor will ask.

What an action layer has to provide #

We'd require four things, and a data access model provides none of them.

A decision before the effect. Every proposed action is evaluated against policy before anything happens, and the evaluation returns pass, warn, review or block. Warn is not a courtesy; it means the action may proceed but something about it is notable and must be recorded. Review means it may not proceed yet: the workflow waits for a named person to approve it, and only the approval releases the effect. Allow-and-deny systems force every ambiguous case into one of two buckets, and the ambiguous cases are the interesting ones.

A corrective path when the answer is no. A block that says only that the action was denied pushes the human into guesswork. A useful decision names what should happen instead, whether a different action, a screen to send the operator to, or a manual step the system cannot perform.

Evidence. Every decision emits a record carrying the decision identifier, the revision of the policy bundle that produced it, and a hash of the inputs it saw. That triple lets you answer, months later, why the system did what it did, using the policy as it existed then. Without the bundle revision an audit judges last quarter's decisions by this quarter's rules and reaches the wrong conclusion confidently.

Durability. An action half-applied when a process dies is the worst outcome available, which is why these workflows run on a durable execution engine rather than in a request handler that a deployment can interrupt.

The shape we'd build #

The agent emits a proposal, never a call. An agent on Agent Runtime returns a typed object naming the action, its parameters and the evidence behind it, and holds no credentials for the system of record, so a prompt injection hidden in a retrieved document has nothing to steal. Structured output makes the proposal a schema check rather than a parse:

agents/servicing/proposal.pypython
from pydantic import BaseModel, Field, model_validator
from typing import Literal

class Proposal(BaseModel):
    action: Literal["issue_refund", "regenerate_payoff_quote", "update_contact"]
    customer_id: str
    amount_usd: float | None = Field(default=None, ge=0)
    reason: str                       # the model's justification, kept as evidence
    evidence_ids: list[str]           # chunk ids or row keys the agent read
    requested_by: str                 # the caller the agent ran on behalf of

    @model_validator(mode="after")
    def refund_needs_amount(self):
        if self.action == "issue_refund" and self.amount_usd is None:
            raise ValueError("issue_refund requires amount_usd")
        return self

A policy evaluator decides, as code. We'd use Open Policy Agent: policies are written as code, reviewed in pull requests, tested in CI, and shipped as a bundle whose revision appears in every decision. Running OPA beside the workflow worker makes evaluation a local call rather than a network dependency that can fail on its own.

policy/refunds.regotext
package actions.refunds

default decision := {"result": "block", "reason": "no rule admitted this action"}

valid_amount if {
  is_number(input.amount_usd)
  input.amount_usd >= 0
}

decision := {"result": "pass"} if {
  input.action == "issue_refund"
  valid_amount
  input.amount_usd <= 100
  count(input.refunds_today_for_customer) == 0
}

decision := {"result": "review", "reason": "above automatic limit, needs a reviewer"} if {
  input.action == "issue_refund"
  valid_amount
  count(input.refunds_today_for_customer) == 0
  input.amount_usd > 100
  input.amount_usd <= 1000
}

Three details in that policy are deliberate. Every admitting rule requires valid_amount, because Rego compares across types and orders null below every number, so without is_number a refund with no amount would satisfy amount_usd <= 100 and pass; we'd keep a test for an omitted amount and an explicit null beside the policy, and the schema refuses both before the policy ever sees them. A refund between $100 and $1,000 returns review, not warn, because the workflow treats warn as permission to execute and review as a hold until someone approves; mixing the two is how a "needs a reviewer" label ends up on a refund that has already left. The daily check sits in both admitting rules, so a second refund for the same customer on the same day falls through to the default block whatever its size. The input carries more than the proposal. It also holds facts the evaluator looks up itself, such as how many refunds this customer already had today, so the policy never trusts the model's account of history.

The tests live beside the policy and run with opa test in CI. These six pass against OPA 1.21, and opa check --strict is clean:

policy/refunds_test.regotext
package actions.refunds_test

import data.actions.refunds

test_small_refund_passes if {
  refunds.decision.result == "pass" with input as {"action": "issue_refund", "amount_usd": 40, "refunds_today_for_customer": []}
}

test_mid_refund_needs_review if {
  refunds.decision.result == "review" with input as {"action": "issue_refund", "amount_usd": 400, "refunds_today_for_customer": []}
}

test_large_refund_blocks if {
  refunds.decision.result == "block" with input as {"action": "issue_refund", "amount_usd": 4000, "refunds_today_for_customer": []}
}

test_null_amount_blocks if {
  refunds.decision.result == "block" with input as {"action": "issue_refund", "amount_usd": null, "refunds_today_for_customer": []}
}

test_missing_amount_blocks if {
  refunds.decision.result == "block" with input as {"action": "issue_refund", "refunds_today_for_customer": []}
}

test_second_refund_today_blocks if {
  refunds.decision.result == "block" with input as {"action": "issue_refund", "amount_usd": 40, "refunds_today_for_customer": ["r-1"]}
}
opa test policy/ -v

A durable workflow executes. We use Temporal. A workflow admitted by policy survives worker restarts, deploys and infrastructure failures, resuming at the step it reached. Every side effect is an activity with its own retry policy and an idempotency key, so a retry after a timeout doesn't issue the refund twice. A team that wants to stay inside Databricks can get part of the way with a Lakeflow Job per action type and Databricks' DurableAgentServer for the agent itself, but a reviewer who takes three days to approve is a workflow state, and a general workflow engine models that more naturally than a job. In Temporal's Python SDK the whole shape is short. It runs against the SDK's time-skipping test server (temporalio 1.34) for a pass, an approved review and a review that expired:

workflows/refund.pypython
# pip install temporalio
import asyncio
from dataclasses import dataclass
from datetime import timedelta

from temporalio import activity, workflow
from temporalio.common import RetryPolicy


@dataclass
class AdmittedRefund:
    decision_id: str      # from the decision record, reused as the idempotency key
    result: str           # "pass", "warn" or "review"; a block never starts a workflow
    customer_id: str
    amount_usd: float


@activity.defn
async def issue_refund(refund: AdmittedRefund) -> str:
    # Call the payments API with refund.decision_id as its idempotency key,
    # so a retry after a timeout returns the first refund instead of a second one.
    return f"refund-for-{refund.decision_id}"


@workflow.defn
class RefundWorkflow:
    def __init__(self) -> None:
        self.approved_by: str | None = None

    @workflow.signal
    def approve(self, reviewer: str) -> None:
        self.approved_by = reviewer

    @workflow.run
    async def run(self, refund: AdmittedRefund) -> str:
        if refund.result == "review":
            try:
                await workflow.wait_condition(
                    lambda: self.approved_by is not None, timeout=timedelta(days=3)
                )
            except asyncio.TimeoutError:
                return "expired without approval"
        return await workflow.execute_activity(
            issue_refund,
            refund,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )

The three-day wait costs nothing while it waits, survives every deploy in between, and the approval arrives as a signal naming the reviewer, which goes into the evidence. We'd start it with the decision id as the workflow id and id_reuse_policy=WorkflowIDReusePolicy.REJECT_DUPLICATE, so a second start for the same decision raises WorkflowAlreadyStartedError, even after the first run has finished. That holds only while the first run's history is retained in the namespace (workflow id reuse policy); after the retention period the same id can start again, so the decision table's unique decision id and the payment provider's idempotency key stay the real guard against a late replay. Without that policy Temporal only refuses a duplicate while the first is still running.

If the agent itself runs on DurableAgentServer (from databricks_agentkit, deployed to Agent Runtime by the Agent Bricks CLI), Preetham Reddy's write-up of a voice agent on Databricks pulls out the three properties we'd lean on, all in Databricks' agent server documentation. Invocations that share a session id run one at a time and in order, and without one an invocation is sessionless. Every invocation carries a client UUID that the server treats as an idempotency key while it keeps the record, so resending the same request returns the existing invocation instead of running the agent again, and reusing an id for a different request returns 409. And a handler registered with @app.recover takes over when a run's heartbeats stop, so a replacement attempt starts within seconds; the server doesn't retry a handler that raised an exception. None of that replaces the policy step; it means the agent half of the pipeline doesn't run twice when a deploy lands mid-turn.

The decision is written down. Every evaluation, whatever its result, becomes a row (below), and the row is written before the effect.

pass

warn

review

approved

block

Agent reads
governed data

Structured proposal

Policy evaluation
OPA bundle

Durable execution
on Temporal

Execute
plus flag

Wait for
human approval

Corrective action
returned

System of record

Decision evidence
id, revision, input hash

Where the proposal stops and the governed pipeline begins

Preetham Reddy, TechFabric's founder, wrote up the voice agents we're building for auto refinance in AutoFabric, our own loan platform on Databricks, and the design is this chapter applied to a phone call. An outbound voice agent is an automated dialer under the TCPA however conversational it sounds, so the action that needs governing is the dial itself. Consent lives in its own governed Unity Catalog table, one row per consent event rather than one per borrower, with the source, timestamp, exact disclosure text and channel, and revocations in the same stream; the current state is always derived.

The gate runs before the dial and the language model has no say in it. It's a deterministic function: consent present and not revoked, the number off the internal and National Do Not Call lists, local time at the called party between 8am and 9pm, attempts within the frequency policy, and the stricter state rules applied. If any check fails, the call never happens. That's the proposal-and-policy split above with the stakes made concrete: the agent can be brilliant on the call and still has no path to a call it wasn't allowed to make.

The same write-up and its companion on decision models in underwriting show the other half. Where a gate needs judgement rather than rules, such as "is this loan package ready to submit", a decision model answers typed questions with a calibrated probability, ordinary code branches on the answer, and low confidence goes to a person. The model's freedom is bounded by the shape of the question, which is what lets it sit in the path of an action.

Where the evidence goes #

We'd write decision records back into the platform, so governance decisions about actions become governed data with the same retention and access controls as everything else. One row per decision is an operational write, so the write path belongs in Lakebase. Synced tables won't get it back out: they copy Unity Catalog tables into Lakebase, the opposite direction. What carries Lakebase writes into Unity Catalog is the Lakebase change data feed, in Public Preview (a workspace admin enables it on the Previews page). It needs Postgres 16 to 18, REPLICA IDENTITY FULL on the table and CAN MANAGE on the project, and it lands each change about every 15 seconds in a managed Delta table named lb_action_decisions_history, with the source columns plus _pg_change_type, _pg_lsn, _pg_xid, _timestamp and a _sort_by ordering key:

lakebase/action_decisions.sqlsql
CREATE TABLE action_decisions (
  decision_id      uuid PRIMARY KEY,
  proposed_at      timestamptz NOT NULL,
  action           text NOT NULL,
  requested_by     text NOT NULL,
  result           text NOT NULL CHECK (result IN ('pass', 'warn', 'review', 'block')),
  reason           text,
  bundle_revision  text NOT NULL,
  input_hash       text NOT NULL,
  workflow_id      text
);

ALTER TABLE action_decisions REPLICA IDENTITY FULL;

Because the table is append-only, every history row should be an insert; an update_postimage or delete in lb_action_decisions_history is itself an audit finding, and we'd alert on it.

An auditor asking which automated refunds were blocked last quarter, and under which policy revision, then gets a SQL query rather than a project. We'd grant SELECT on the history table to the audit group and nothing at all to the agent's identity: the agent should be able to propose, never to read or rewrite the record of what was decided about it.

Advanced: what a mature action layer does #

An allow-or-deny check in front of a tool call is where a first version stops. What follows is what we'd add before defending the layer in a regulated audit.

Simulation against historical proposals. A bundle can be evaluated against a recorded stream of past proposals without executing anything, producing a diff of which decisions would change. That turns a policy edit from a leap of faith into a reviewable artefact, and it is what makes people willing to change policy at all.

Two-key and quorum actions. Some effects should require a second human, or a second system, above a threshold. Encoding that in policy rather than in application code makes the threshold versioned evidence rather than a constant somebody edited.

Compensation rather than rollback. A refund that has left the building cannot be rolled back, only compensated by a later action. Modelling every action with its compensating counterpart, inside one durable workflow, is what lets a multi-step business transaction fail safely at step four.

Time and rate scoping. Policies referencing how many actions of a kind were already taken today, by this agent, for this customer, catch the failure mode per-action checks miss entirely, which is a thousand individually reasonable actions.

Decision replay. Because the evidence record carries the bundle revision and an input hash, any historical decision can be re-evaluated against the current bundle to see whether today's rules would decide differently. That is how you answer a regulator asking whether a past harm is still possible.

Governing effects, not just access

Capability Databricks, with an action layer added AWS Azure GCP
Who may read the data Unity Catalog, one model IAM plus Lake Formation Entra ID plus Purview IAM plus Dataplex
Whether an action may be taken Not built in; UC function EXECUTE grants, plus a policy evaluator you add IAM grants API permissions to principals, not decisions about effects Not addressed Not addressed
Evidence tied to policy version Your decision table, in Lakebase and carried to Unity Catalog by its change data feed CloudTrail records the call, not the policy revision behind it Activity log, same limitation Audit Logs, same limitation
Durable multi-step execution Lakeflow Jobs or DurableAgentServer, or Temporal beside the platform Step Functions, durable, no policy layer Durable Functions or Logic Apps Workflows
Decisions queryable beside business data Written into the lakehouse Export and join it yourself Export and join it Export and join it

No platform in that table, Databricks included, evaluates a proposed business action for you; what differs is where the evidence lands. IAM, Lake Formation, Purview and Dataplex are good at what they do, and Step Functions is an excellent durable engine we'd happily use. The narrow claim is that none of them evaluates a proposed business action against versioned policy and emits evidence of the evaluation, because it is not what they are for. IAM answers whether a principal may call an API, not whether this refund, for this customer, at this amount, is something the business agreed to do automatically.

The design review test #

In a design review we'd ask three questions of every call that leaves the platform and changes something, and a call site that cannot answer all three is not ready for production.

none

named bundle

none

decision record

no

idempotent activity

Effectful call

Which policy
admitted it

Move behind
the action pipeline

What evidence
was written

Safe if the
process dies

Ready for production

The three questions every effectful call has to answer

The technology is replaceable, since Open Policy Agent could be another evaluator and Temporal another durable engine without changing anything argued here. The boundary itself is not replaceable, and neither is the discipline of keeping every effect on the far side of it.