Analysing and serving · Chapter 17
MLflow on Databricks: tracking, model registry and serving
Reviewed by TechFabric engineers. How we write this guide →
MLflow on Databricks is managed MLflow 3: experiment tracking, a model registry that lives in Unity Catalog (models get three-level names like tables, the same grants and lineage, and aliases instead of stages), Model Serving endpoints, and tracing and evaluation for generative applications. On MLflow 3 the registry defaults to Unity Catalog, so mlflow.set_registry_uri("databricks-uc") is only needed on older clients. The current release on PyPI is 3.17.2, from 21 October 2026.
This chapter is the path we'd set up for a model that makes decisions someone may have to defend: train from a pinned table version, register with a signature, promote by alias through an approval, serve a version, and keep the record. The code runs on Databricks Runtime ML with mlflow>=3.0, and the GenAI parts need mlflow[databricks]>=3.14.0.
The short version
A model is a claim that some data, processed some way, produces a number your business acts on. Six months later somebody will ask why a customer was declined, and the useful answer names the model version that scored them, the table version that trained it, who approved it and what it measured. MLflow on Databricks records all four in the same catalog that governs your tables, as long as you train from a pinned table version, log that input, and make approval a step with a permission rather than a chat message.
What changed in MLflow 3 #
If you learned MLflow on version 2, these are the differences you'll hit first. Sources: MLflow 3 for models and models in Unity Catalog, both updated 9 October 2026.
| Area | MLflow 2 habit | MLflow 3 on Databricks |
|---|---|---|
| Logging a model | artifact_path="model", inside an active run |
name="model"; artifact_path still works but is deprecated, and no active run is required |
| Model object | Pinned to the run that produced it | A LoggedModel that carries metrics, parameters and traces across runs and environments |
| Registry | Workspace registry, stages (Staging, Production) |
Unity Catalog by default; stages aren't supported, use aliases and catalogs per environment |
| Classic evaluation | mlflow.evaluate |
mlflow.models.evaluate; mlflow.evaluate is deprecated |
| GenAI evaluation | Not available | mlflow.genai.evaluate with scorers, over traces |
| Promotion workflow | Webhooks or a hand-built job | Deployment jobs (Public Preview): evaluation, approval and deployment as one Lakeflow job |
| Prompts | Strings in code | Prompt registry in Unity Catalog (Beta) |
New model versions in Unity Catalog must have a signature. Passing an input_example to log_model infers one, which is the easiest way to never meet that error.
The four things MLflow does #
Tracking records each training run's parameters, metrics, code version and inputs, so a result can be rebuilt a month later. Models is the packaging format: a trained model saved with its dependencies, a signature and a flavour (scikit-learn, PyTorch, a generic Python function) so it can be served without knowing how it was built. The registry, in Unity Catalog, holds immutable numbered versions and movable aliases. We'd have every application load models:/<name>@champion, never a version number, so a rollback is moving an alias back rather than redeploying a service. A serving endpoint is the one exception, because it names a version (see below). Evaluation scores a model against a held-out set, with mlflow.models.evaluate for classic models and mlflow.genai.evaluate for generative ones.
Train from a pinned table version and log it as an input #
Lineage from a model back to its training table is recorded when the run logs that table as an input (MLflow 2.11.0 or later). The edge is only useful if the version it points at still exists, so we'd read an explicit Delta version and log the same one:
import mlflow
from sklearn.calibration import calibration_curve
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import brier_score_loss, roc_auc_score
from sklearn.model_selection import train_test_split
TABLE = "originations_prd.gold.application_features"
MODEL = "originations_prd.models.application_risk"
mlflow.set_experiment("/Shared/originations/application_risk") # absolute path, shared by every run
version = spark.sql(f"DESCRIBE HISTORY {TABLE} LIMIT 1").first()["version"]
dataset = mlflow.data.load_delta(table_name=TABLE, version=version)
pdf = dataset.df.toPandas()
X = pdf.drop(columns=["application_id", "defaulted"])
y = pdf["defaulted"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=7)
with mlflow.start_run(run_name="risk_hgb"):
mlflow.log_input(dataset, context="training") # this is the lineage edge
params = {"max_depth": 6, "learning_rate": 0.05}
model = HistGradientBoostingClassifier(**params).fit(X_train, y_train)
prob = model.predict_proba(X_test)[:, 1]
mlflow.log_params(params)
mlflow.log_metrics({
"auc_test": roc_auc_score(y_test, prob),
"brier_test": brier_score_loss(y_test, prob),
})
observed, predicted = calibration_curve(y_test, prob, n_bins=10, strategy="quantile")
mlflow.log_table(
{"mean_predicted": predicted.tolist(), "observed_rate": observed.tolist()},
artifact_file="calibration_by_decile.json",
)
info = mlflow.sklearn.log_model(
sk_model=model,
name="model",
input_example=X_train.head(5), # infers the signature Unity Catalog requires
registered_model_name=MODEL,
)
print(MODEL, "version", info.registered_model_version)The registry page uses mlflow.data.load_delta the same way. After registration, the model version's Lineage tab shows the table, and Catalog Explorer shows the model downstream of it.
Why the calibration table is in that run #
When a model's output is a probability that drives a threshold (approve, refer, decline), the AUC says it ranks well and says nothing about whether 0.8 means 80%. We'd set thresholds from your own calibration curve, which is the design our write-up on decision models in auto refinance underwriting recommends for AutoFabric, our loan platform on Databricks: log every request, answer, confidence score, model version and timestamp to a Delta table, join lender outcomes on the application ID, and compare predicted against observed rates by confidence decile. The calibration_by_decile.json above would be the offline half of that curve, and the inference record joined to outcomes the online half, which is why the serving section below cares where requests are logged.
Promote by alias, approve by permission #
A version number is immutable. An alias is a movable label such as champion or challenger. Batch jobs and notebooks should load by alias, so a rollback is moving the label back:
import mlflow
from mlflow import MlflowClient
client = MlflowClient()
client.set_registered_model_alias(name="originations_prd.models.application_risk", alias="challenger", version=7)
scorer = mlflow.pyfunc.load_model("models:/originations_prd.models.application_risk@champion")Environments are catalogs, not stages. Copying an approved version from a dev catalog into prod keeps the two registries separate (MLflow 2.8.0 or later):
src = "models:/dev.ml.application_risk/7"
copied = client.copy_model_version(src, "originations_prd.models.application_risk")
client.set_registered_model_alias("originations_prd.models.application_risk", "champion", copied.version)The permissions decide who can do what, and they're the part we'd design first. Creating a model needs CREATE MODEL on the schema; adding versions needs ownership or CREATE MODEL VERSION on the model; loading by alias needs EXECUTE; setting or deleting an alias needs ownership of the model. Registered models are a function subtype, so grants use ON FUNCTION:
GRANT USE CATALOG ON CATALOG originations_prd TO `ml-engineers`;
GRANT USE SCHEMA, CREATE MODEL ON SCHEMA originations_prd.models TO `ml-engineers`;
GRANT USE CATALOG ON CATALOG originations_prd TO `batch-scoring`;
GRANT USE SCHEMA ON SCHEMA originations_prd.models TO `batch-scoring`;
GRANT EXECUTE ON FUNCTION originations_prd.models.application_risk TO `batch-scoring`; -- loading by alias needs all threeDeployment jobs #
A deployment job (Public Preview, updated 11 September 2026) is a Lakeflow job connected to a registered model that runs whenever a new version is created. The templates are an evaluation task, an approval task and a deployment task. The approval mechanism is a Unity Catalog tag: a task whose name starts with approval passes only when the model version carries that tag set to Approved, so its first run fails by design, and an approver clicks Approve on the version page, which sets the tag and repairs the run. Approving needs APPLY TAG on the model and CAN MANAGE RUN on the job, which fits the split above.
The job triggers itself with the model owner's credentials, and its evaluation task runs code that came with the new version. Databricks' warning is blunt about the consequence: granting CREATE MODEL VERSION lets that person execute arbitrary code as part of the job. Every task in a job runs as the job's one Run As identity, so the approval task can't make the deployment task safer than the evaluation code that ran before it. We'd split it into two jobs. The deployment job connected to the model evaluates and stops at approval, running as a service principal created for it with read access to the evaluation data and the model and nothing else: it can't apply the approval tag, move an alias, edit the serving endpoint or trigger the second job. Promotion is a separate job, run as a different principal that holds the alias and endpoint rights, triggered only by a person in the risk-owner group after they approve, and it runs no code that came with the model version. Connecting the evaluation job from code:
client.update_registered_model("originations_prd.models.application_risk", deployment_job_id="<job-id>")Advanced: what production actually demands #
Getting a governed model into production is the first half. Running it is the second, and it has a bill and a record to keep. Endpoints bill on provisioned capacity and uptime rather than the query patterns your finance team has learned to forecast, and a rarely used endpoint left running is pure waste. Scale to zero where the caller can tolerate a cold start, and read Cost and performance before committing to always-on capacity.
Serving a version #
Model Serving runs a registered version behind an authenticated, serverless endpoint. A served entity names a version number, so moving an alias doesn't change what an endpoint serves; the deployment task, or your CI, updates the endpoint config. With the Databricks SDK for Python:
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import EndpointCoreConfigInput, ServedEntityInput
w = WorkspaceClient()
w.serving_endpoints.create_and_wait(
name="application-risk",
config=EndpointCoreConfigInput(
served_entities=[
ServedEntityInput(
entity_name="originations_prd.models.application_risk",
entity_version="7",
workload_size="Small",
scale_to_zero_enabled=True, # set False where a cold start breaks the caller's timeout
)
]
),
)That shape follows the create endpoint page (updated 9 October 2026). An endpoint can serve two versions with a traffic split for a canary; shadow scoring, where the challenger sees a copy of traffic and its output isn't acted on, is something you build in the caller or in batch over logged requests.
Where the inference record lives #
There are two places now, and we'd choose deliberately. Inference tables are configured per endpoint, so coverage is whatever each endpoint's owner remembered to switch on. The unified trace table (Beta, updated 14 October 2026) captures every request and response across Unity Gateway services into one Unity Catalog table in OpenTelemetry format, and Databricks recommends it for new deployments. Three details matter. Only the metastore admin who creates it can read it until someone grants access, so the GRANT SELECT to monitoring and security groups belongs in the same change. Delivery isn't guaranteed for requests that ended in 401, 403, 429 or 500, which are the requests an incident review wants. And it complements the audit log, which stays the record of who invoked what. Our working rule: the trace table for debugging and cost across everything, and per-endpoint inference tables kept on any model whose decisions you'd have to defend until the trace table leaves Beta.
Drift monitoring should watch inputs as well as outputs. A shift in applicant income or vehicle age tells you the world moved before the model's errors pile up, and Observability covers where that belongs. Deprecating a model with live consumers is harder than deploying one; the inference record tells you who still calls an endpoint, which turns a deprecation from a negotiation into a schedule.
MLflow 3 and tracing for generative systems #
A trace is the recorded execution of one request through your application, held as a tree of spans: each model call, retrieval or tool call with its inputs, outputs, timing and status. Autologging covers more than 30 libraries with one line (mlflow.openai.autolog(), mlflow.langchain.autolog()), and @mlflow.trace adds your own functions to the same tree. With a trace, a wrong answer becomes a step you can point at: retrieval returned nothing relevant, or it did and the model ignored it, or a tool got the loan number in the wrong format. Those are three faults with three fixes, and from outside they look identical.
Store traces in Unity Catalog from the first experiment #
By default traces live in the experiment, which caps at 100,000 traces per experiment. Unity Catalog trace storage (updated 16 October 2026, MLflow 3.14.0 or later) writes them to Delta tables you can query in SQL and grant like any table, with no per-experiment cap:
# %pip install --upgrade "mlflow[databricks]>=3.14.0"
import mlflow
from mlflow.entities.trace_location import UnityCatalog
spark.sql("CREATE SCHEMA IF NOT EXISTS originations_prd.agent_traces")
mlflow.set_experiment(
"/Shared/servicing/assistant",
trace_location=UnityCatalog(
catalog_name="originations_prd",
schema_name="agent_traces",
table_prefix="servicing",
),
)Unity Catalog traces are read back through a SQL warehouse, so the docs' setup also sets os.environ["MLFLOW_TRACING_SQL_WAREHOUSE_ID"] = "<warehouse-id>" before the experiment is created, and whoever reads traces needs CAN USE on that warehouse.
That creates servicing_otel_spans, servicing_otel_logs, servicing_otel_metrics and servicing_otel_annotations. The limits to plan around: an experiment can only be bound to a trace location when it's created, so decide on day one; writers need MODIFY and SELECT on all four tables, and ALL PRIVILEGES isn't enough; ingestion starts at 200 traces a second per workspace and 100 MB a second per table; single traces can't be deleted from the UI or SDK, only with SQL DELETE; and traces over 25 MB can't be opened in the UI. Trace storage can't yet target default storage or Private Link-protected storage.
A trace holds whatever a customer typed and everything retrieved for them, so we'd decide retention when tracing is switched on: full traces for a short window where debugging happens, a reduced set kept longer for trends and evaluation, and the same restrictive grants you'd put on any table of customer text.
Evaluate with scorers, then reuse them in production #
mlflow.genai.evaluate runs your app over an evaluation set, traces each call, and scores the traces. Built-in judges include Safety, RelevanceToQuery, RetrievalGroundedness, Correctness and Guidelines (evaluation tutorial, updated 9 October 2026):
import mlflow
from mlflow.genai.scorers import Correctness, Guidelines, RelevanceToQuery, Safety
@mlflow.trace
def answer(question: str) -> str:
chunks = retrieve_policy(question, k=5) # your retrieval
return generate(question, chunks) # your model call
eval_set = [
{
"inputs": {"question": "Can a customer defer one payment after a hardship call?"},
"expectations": {"expected_facts": ["one deferral per 12 months", "hardship must be documented"]},
},
]
results = mlflow.genai.evaluate(
data=eval_set,
predict_fn=answer,
scorers=[
Correctness(),
RelevanceToQuery(),
Safety(),
Guidelines(name="cites_clause", guidelines="The answer names the policy clause it relies on."),
],
)The keys in inputs are the arguments of predict_fn. The evaluation set should be written by the people who'll use the system, which Agents on Databricks argues at length, and the same scorers can then run over production traces so the definition of quality doesn't change when the system ships.
Prompts are versioned artefacts too #
A prompt revision is a production change. The prompt registry (Beta) stores prompts in Unity Catalog with versions and aliases, and needs CREATE FUNCTION, EXECUTE and MANAGE on the schema:
import mlflow
prompt = mlflow.genai.register_prompt(
name="originations_prd.agent_traces.servicing_answer",
template="Answer using only the policy excerpts below.\n\n{{excerpts}}\n\nQuestion: {{question}}",
)
mlflow.genai.set_prompt_alias("originations_prd.agent_traces.servicing_answer", alias="production", version=prompt.version)
live = mlflow.genai.load_prompt("prompts:/originations_prd.agent_traces.servicing_answer@production")
text = live.format(excerpts="...", question="...")We'd register the prompt in the same release as the agent that uses it, so an incident can name the prompt version, the model version and the trace together.
Training on serverless GPUs #
AI Runtime, Databricks' serverless GPU compute, logs to MLflow too: its .distributed() API creates MLflow runs automatically, and you customise the experiment by setting it with an absolute path (migration guide, updated 21 October 2026). Set it from the first run so GPU training feeds the same lineage as everything else. Our AI Runtime note covers where we'd train on it.
Running a governed model lifecycle on each platform
| Capability | Databricks | AWS | Azure | GCP |
|---|---|---|---|---|
| Experiment tracking | MLflow 3, managed | SageMaker Experiments, or self-hosted MLflow | Azure ML jobs, MLflow compatible | Vertex AI Experiments |
| Registry identity | A Unity Catalog object | SageMaker Model Registry, separate | Azure ML registry, separate | Vertex AI Model Registry, separate |
| One permission model for data and models | Yes, the same grants | Lake Formation and IAM, reconciled by you | Purview and Azure ML roles | Dataplex and Vertex IAM |
| Lineage to the training table version | Recorded when the run logs its input | Assemble it | Assemble it | Assemble it |
| Request and response record | Inference tables or the unified trace table, queryable in SQL | Data capture to S3, wire it up | Wire it up to Log Analytics | Wire it up to BigQuery |
These are capable products. SageMaker's training and tuning tooling is strong, and some teams prefer Vertex pipelines. The difference that matters for an audit is narrower: on Databricks the model and the table that trained it share one permission and lineage system, and elsewhere your team builds that link and then has to defend it.
What's popular