# Databricks job completion triggers for Lakeflow Jobs

> In Beta since 3 November 2026, a Lakeflow Job can start when upstream jobs finish, including other workspaces on the same Unity Catalog metastore.

By Sushruth Aeluguri, 11 November 2026. Databricks Field Guide, TechFabric. Canonical: https://databricks.techfabric.com/notes/trigger-lakeflow-jobs-when-upstream-jobs-complete-beta/

We'd refuse a pull request that swaps the publish-margin job onto a job completion trigger. The [orchestration chapter](https://databricks.techfabric.com/guide/orchestration/) waits on `main.silver.orders` and `main.silver.shipments` for five minutes after the last commit, and it will not start again inside fifteen minutes, with one run at a time. That quiet period is what makes a burst of silver commits one gold run. The same chapter shows two retries, a five-minute gap and a thirty-minute timeout already using an hour and forty minutes, so a table due at seven after upstream lands at five is late on that arithmetic alone. [Job completion triggers](https://docs.databricks.com/aws/en/jobs/job-completion-triggers) are in Beta as of the [3 November 2026 release notes](https://docs.databricks.com/aws/en/release-notes/product/2026/november#trigger-lakeflow-jobs-when-upstream-jobs-complete-beta): the downstream job names the upstream jobs, including jobs in other workspaces on the same Unity Catalog metastore, and nobody edits the upstream definitions. Keep the [table update trigger](https://docs.databricks.com/aws/en/jobs/trigger-table-update) on publish-margin. Use the Beta when the upstream work leaves no table, an export or a notify another team deploys.

A [Run Job task](https://docs.databricks.com/aws/en/jobs/tasks/run-job) stays the graph when one team owns both jobs in the same workspace.

> **Our position**: Leave publish-margin on its table update trigger. Add a job completion trigger on a downstream job whose upstream work leaves no table, including a job another team deploys in another workspace on the same metastore, and hold the publish when any upstream `termination_code` is `SUCCESS_WITH_FAILURES`.

## The check before publish

Hold a partial success. The default filter will not.

Succeeded (`SUCCESS`) matches `SUCCESS_WITH_FAILURES`, so a run that ended with failures still starts the downstream job ([job completion triggers](https://docs.databricks.com/aws/en/jobs/job-completion-triggers)). Two finishes of orders before shipments finishes once become one downstream run, and it keeps the later orders completion. We'd pass `{{job.trigger.job_completion.completed_jobs}}` in as `upstream_completions` and publish only when `publish_ready` says so. The parameter and the codes are on the [job completion page](https://docs.databricks.com/aws/en/jobs/job-completion-triggers).

```python
def publish_ready(completions):
    """completions is the upstream_completions parameter.
    [] is a manual run. The default Succeeded filter also matches SUCCESS_WITH_FAILURES."""
    if not completions:
        return "hold"
    if any(job["termination_code"] != "SUCCESS" for job in completions):
        return "hold"
    return "publish and store each run_id"
```

Store the `run_id` next to the gold table. The Run as service principal needs a seat in every upstream workspace and CAN VIEW on each watched job, and the preview has to be enabled in those workspaces as well as the downstream one, all on one metastore ([job completion triggers](https://docs.databricks.com/aws/en/jobs/job-completion-triggers)). We'd run Test trigger and stop while it reports a grant. The freshness check on the gold table stays, the same one the [orchestration chapter](https://databricks.techfabric.com/guide/orchestration/) already wants, because a trigger that never starts sends no failure mail. We'd treat a principal missing from the orders workspace as that miss. Turning the trigger on after lunch does not replay the morning: historical completions do not start a run.

```mermaid Which wait belongs on the downstream job
flowchart TD
  A["Another job must finish first"] --> B{"One team owns both jobs in one workspace"}
  B -->|"Yes"| C["Run Job task in the parent graph"]
  B -->|"No"| D{"Downstream watches tables the upstream job writes"}
  D -->|"Yes"| E["Table update trigger"]
  D -->|"No"| F["Job completion trigger on the downstream job"]
```

A model promotion stays a [deployment job](https://docs.databricks.com/aws/en/mlflow/deployment-job), and the [MLflow chapter](https://databricks.techfabric.com/guide/mlflow/) keeps a person on the approval tag. A stream that has to stay up stays in [real-time and CDC](https://databricks.techfabric.com/guide/real-time-and-cdc/).
