An expensive join that feeds three outputs in a serverless notebook either runs three times or gets written to a scratch table somebody has to remember to drop. The release notes for 23 October 2026 put DataFrame caching on serverless compute in Public Preview, so df.cache() or df.persist() now reuses that result for the rest of the session. It needs serverless environment version 5 or above, Databricks Connect for Python 18.1.8 or Databricks Connect for Scala 18.1.5 and up, and it's on by default for most workspaces. The call keeps its classic name and changes what it means, which is what we'd want every team to understand before leaning on it.
How it behaves differently from a classic cluster #
The serverless caching documentation describes an eager cache. cache() runs the query before it returns and materializes the result as a snapshot, and only the cached DataFrame and the DataFrames derived from it read that snapshot. An identical query built again from the source tables reads the tables. On classic compute any query with the same plan shape picks up the cache, and cached data is refreshed in some cases. On serverless the snapshot is fixed and private to the session, and unpersist() destroys the DataFrame and everything derived from it, so the next query on any of them raises CACHE_DATAFRAME_ACCESS_AFTER_UNCACHE.
We think these are better semantics for production code. A snapshot that can't drift makes a multi-step analysis repeatable, and an eager cache fails at the cache() line, where the stack trace makes sense.
The price is two classic habits that break quietly. A notebook that rebuilt the same DataFrame in another cell and relied on plan matching now recomputes it, which shows up only as a slower run. A notebook that wrote to a source table and then read the cached DataFrame expecting the new rows now gets the old ones. Before we'd move a classic notebook over, we'd search it for both patterns, keep one cached variable to derive from, and rebuild from the source after any write.
The snapshot keeps what you could read #
This is the section we'd take to whoever owns Unity Catalog policy. The documentation on isolation and privileges says cached data contains what the user was allowed to read when they cached it, and that permission changes, row filters and column masks applied to the source tables afterwards don't apply to it. Any user who can run code on serverless compute can cache, with no extra privilege. Only the session that made the snapshot can read it, which keeps the exposure small, but a revoked grant doesn't reach a session that cached the rows before the revocation. On a table behind the row filters and column masks a regulated customer depends on, we'd have jobs call unpersist() as soon as the last step that reads the snapshot has run. Where the snapshot lives matters to the same people. A serverless workspace keeps it in default storage under the managed services key, if one is configured, and a classic workspace keeps it in the workspace storage bucket. We'd add it to the map of stores a key does and doesn't cover, from customer-managed keys.
The preview isn't available in workspaces with the compliance security profile, or on serverless compute in AWS GovCloud, per the requirements and limitations.
What we'd cache, and what we'd write to a table #
We'd follow the documentation's best practices and cache after a join, aggregation or filter that several later steps read, and never a plain table read, which only copies the table. The two uses we'd reach for first are a foreachBatch function that writes one micro-batch to several sinks, and training or tuning several models on the same prepared features. Aggregates trip one limitation, because a column name with spaces or parentheses, like the sum(amount) an aggregation produces, can't be cached. We'd alias those columns before calling cache().
A snapshot lasts until the session ends, seven days at most, and after that a cached DataFrame raises an error instead of reading the source tables again. So a long-running job that caches a lookup once at startup is the wrong design, and we'd write that result to a table, as we would anything another notebook, job run or user needs. In SQL the documented route is a session-local temporary table, because CACHE TABLE isn't supported on serverless compute. With caching available, fewer development notebooks need an all-purpose cluster, which fits the default we give serverless in choosing the right compute.