Databricks Field Guide

Databricks real-time mode for Lakeflow pipelines reaches GA

Reviewed by TechFabric engineers. How we write this guide →

A scheduled pipeline that writes a Delta table doesn't become a millisecond pipeline because this mode is generally available. We'd leave it on its schedule. Real-time mode in Lakeflow pipelines is generally available on classic and serverless compute as of 29 October 2026, and Delta is still not a source or a sink it can use.

We'd turn the mode on only when the output is a decision written to Kafka, Event Hubs or a ForEach writer, and we want the pipeline's recovery around a query that a direct Structured Streaming job has been able to run since 19 March 2026. The real-time chapter already draws that line for a direct query. Lakeflow pipelines are the Databricks product this trigger now runs in, built on Apache Spark Declarative Pipelines, which is the name on the announcement. The glossary is where those names sit.

We haven't run this mode on a Lakeflow pipeline.

Leave the Delta pipeline on micro-batch #

Take a continuous pipeline whose job is to land rows in a Delta table. The connector table is what makes the October date a non-event for it. Delta is unsupported on both sides. A stream-to-stream join is unsupported too, so a pipeline that needs one stays on micro-batch even if the sink would otherwise qualify. A ForEach sink can write each row to a store such as Lakebase or Redis, which the 29 October post states as available now. A native Lakebase sink, with batching, retries and Unity Catalog authentication, is listed there as what comes next. A new design that needs the store now uses ForEach. It doesn't wait on the native sink, and the scheduled table load doesn't need either one.

A second real-time flow in the same pipeline competes for task slots on classic compute and on the current serverless channel, and the latency goes up. Multiple flows without that contention are in Beta, and only on serverless pipelines that use the preview channel. Until that Beta is where we'd run it, the second flow gets its own pipeline.

No

Yes

No

Yes

Lakeflow pipeline output

Must a system act in milliseconds

Keep micro-batch and Delta

Sink is Kafka, Event Hubs or ForEach

Continuous pipeline, RealTime trigger

When a Lakeflow pipeline should use real-time mode

Count the slots before you quote five milliseconds #

The five-millisecond line is a floor, and we'd count task slots before quoting it. Classic compute schedules every stage at once. The sizing table puts eight source partitions at 8 task slots, and the same source plus a shuffle of 20 at 28. That sum is the estimate. A cluster short of slots doesn't inherit the floor. The checkpoint interval defaults to five minutes, which is how often state and offsets are saved, so a longer interval replays more of a failed batch. The flow has to be an update flow with its trigger set to RealTime, on a pipeline that's already continuous. That configuration is on the setup page.

The October post says the Databricks Feature Store reaches a p99 of 200 ms from Kafka to its online feature store on this architecture. That's their query and their store. Read e2eLatencyMs on the flow in front of you, and don't quote the floor or the 200 ms as the latency a Delta table load would hit.

Next

Databricks serverless cost controls: who can run, and how fast