We'd refuse a pull request that swaps the publish-margin job onto a job completion trigger. The orchestration chapter waits on main.silver.orders and main.silver.shipments for five minutes after the last commit, and it will not start again inside fifteen minutes, with one run at a time. That quiet period is what makes a burst of silver commits one gold run. The same chapter shows two retries, a five-minute gap and a thirty-minute timeout already using an hour and forty minutes, so a table due at seven after upstream lands at five is late on that arithmetic alone. Job completion triggers are in Beta as of the 3 November 2026 release notes: the downstream job names the upstream jobs, including jobs in other workspaces on the same Unity Catalog metastore, and nobody edits the upstream definitions. Keep the table update trigger on publish-margin. Use the Beta when the upstream work leaves no table, an export or a notify another team deploys.
A Run Job task stays the graph when one team owns both jobs in the same workspace.
The check before publish #
Hold a partial success. The default filter will not.
Succeeded (SUCCESS) matches SUCCESS_WITH_FAILURES, so a run that ended with failures still starts the downstream job (job completion triggers). Two finishes of orders before shipments finishes once become one downstream run, and it keeps the later orders completion. We'd pass {{job.trigger.job_completion.completed_jobs}} in as upstream_completions and publish only when publish_ready says so. The parameter and the codes are on the job completion page.
def publish_ready(completions):
"""completions is the upstream_completions parameter.
[] is a manual run. The default Succeeded filter also matches SUCCESS_WITH_FAILURES."""
if not completions:
return "hold"
if any(job["termination_code"] != "SUCCESS" for job in completions):
return "hold"
return "publish and store each run_id"Store the run_id next to the gold table. The Run as service principal needs a seat in every upstream workspace and CAN VIEW on each watched job, and the preview has to be enabled in those workspaces as well as the downstream one, all on one metastore (job completion triggers). We'd run Test trigger and stop while it reports a grant. The freshness check on the gold table stays, the same one the orchestration chapter already wants, because a trigger that never starts sends no failure mail. We'd treat a principal missing from the orders workspace as that miss. Turning the trigger on after lunch does not replay the morning: historical completions do not start a run.
A model promotion stays a deployment job, and the MLflow chapter keeps a person on the approval tag. A stream that has to stay up stays in real-time and CDC.