DataFrame caching on serverless compute
DataFrame caching on serverless compute is a Databricks compute / bi capability, introduced October 2026.
Caching a DataFrame with df.cache() or df.persist() on serverless compute, which computes it once and saves the result as a session-local snapshot in cloud storage to reuse for the rest of the session.
- Ask for MEMORY_ONLY and you get disk anyway - serverless accepts any storage level, ignores it, and reports DISK_ONLY once the DataFrame is cached.
- It is eager, not lazy - the DataFrame is computed the moment you call cache(), not on the next action.
- The snapshot is private and perishable - only the session that cached it can read it, and it lasts until the session ends, for at most seven days.
Limitations: Public Preview, on by default. Not available with the compliance security profile or in AWS GovCloud; needs serverless environment version 5 or above (Databricks Connect Python 18.1.8+ or Scala 18.1.5+); CACHE TABLE and the spark.catalog cache methods are unsupported; DataFrames with duplicate or special-character column names, interval columns, or no columns cannot be cached; the persist storage level and the unpersist blocking flag are ignored.
Open in REbricked →- Category
- Compute / BI
- Also known as
- df.cache(), df.persist(), serverless DataFrame cache
- Verified
- 2026-10-26