# Databricks Lakeflow pipeline system tables are GA: cost and refreshes

> The Lakeflow pipeline system tables are generally available. How we would use them for pipeline spend by name and full refresh audits.

By Sushruth Aeluguri, 22 October 2026. Databricks Field Guide, TechFabric. Canonical: https://databricks.techfabric.com/notes/lakeflow-pipelines-system-tables-are-now-generally-available/

A pipeline changes teams halfway through the month. Who gets the bill? We'd require the report owner to answer that before replacing a chargeback collector with the Lakeflow system tables, which Databricks made [generally available on 21 October 2026](https://docs.databricks.com/aws/en/release-notes/product/2026/october#lakeflow-pipelines-system-tables-are-now-generally-available-ga).

Our recommendation is to keep a current-responsibility view for operations and a separate historical view for chargeback, because the person who can fix a pipeline today may have had no responsibility for the usage finance is asking them to explain. Combining them would save a view and create an argument over every transfer.

We haven't validated these tables against a TechFabric billing estate.

## A current inventory and a historical bill ask different questions

Take this worked example, not a customer account. A pipeline's team tag changes from Finance to Platform during the billing period. Both teams agree that costs before the transfer belong to Finance.

| Report choice | What happens to usage before the transfer |
| --- | --- |
| Join all usage to the latest team tag | Platform inherits Finance's earlier usage |
| Join usage to the metadata valid at that time | Finance keeps its share, subject to the agreed transfer rule |
| No matching metadata | Keep the usage in an unassigned bucket for investigation |

The [reference](https://docs.databricks.com/aws/en/admin/system-tables/jobs) describes `system.lakeflow.pipelines` as a slowly changing dimension. That gives a report metadata history to work with. It doesn't decide whether a tag means budget owner, operator or something else. Agree on the tag's meaning first, then reconcile one transferred pipeline, retaining `workspace_id` with `pipeline_id` and checking that the join neither loses nor multiplies usage. A report with more named pipelines but a different total has failed this test. Our [cost chapter](https://databricks.techfabric.com/guide/cost-and-performance/) explains why attribution needs that accounting discipline.

```mermaid How the reporting question changes the pipeline metadata join
flowchart TD
  A["Pipeline usage with workspace and pipeline IDs"] --> B{"What does the report need"}
  B -->|"Current responsibility"| C["Latest pipeline metadata"]
  B -->|"Responsibility at usage time"| D["Metadata valid at that time"]
  C --> E["Current operating inventory"]
  D --> F["Historical chargeback report"]
```

## A refresh count needs an update boundary

The other table, `system.lakeflow.pipeline_update_timeline`, introduces a separate counting problem. The [reference](https://docs.databricks.com/aws/en/admin/system-tables/jobs) splits long updates into hourly slices and puts the final result on the last slice. A row count is therefore the wrong acceptance criterion for a refresh audit.

Choose an update long enough to cross an hourly boundary. In the proposed report, it should appear once, with its slices reconciled and its final result preserved. Only then compare its full-refresh selection with the known update. Getting the total right while losing the update's identity would leave the operator with nothing to investigate.

## Keep the existing inventory until the comparison explains its gaps

Both tables are regional, with a documented 365-day free retention period. Records typically arrive within an hour, and new workspaces can take longer. Those are reasons to keep immediate alerting separate and to arrange storage for older audit history.

Before retiring a collector, the report owner should sign off on the transferred pipeline and the long update. Keep unexplained usage visible. If a region is missing, mark the report incomplete rather than turning that absence into zero spend.