Real-Time Mode
Real-Time Mode is a Databricks data engineering capability, introduced July 2025.
A trigger type for Apache Spark Structured Streaming that swaps periodic micro-batches for long-running batches processed as data arrives, up to 100x faster than micro-batch mode, for operational workloads such as fraud detection on the same Spark APIs.
- Its route to millisecond latency is, paradoxically, longer batches: each one runs for five minutes by default, processing data the moment it lands instead of waiting for the next micro-batch.
- The fastest streaming mode on Databricks asks you to switch off Databricks' own fast engine - the setup checklist says to turn off Photon (and autoscaling, and spot instances).
- It did not stay proprietary - the stateless half of it ships in open-source Apache Spark 4.1.
Limitations: Direct Structured Streaming queries need classic compute on Databricks Runtime 16.4 LTS and above (dedicated access mode, or standard access mode for Python only) with autoscaling, Photon, and spot instances turned off; serverless notebooks and jobs can't run them, so serverless real-time processing goes through Lakeflow pipelines instead. Update output mode only. Sources are Kafka, Event Hubs (Kafka connector), Kinesis, and AWS MSK; Delta is supported as neither a source nor a sink. Stream-to-stream outer joins aren't supported, and stream-to-table joins need a broadcast table of 10 MB or less.
Open in REbricked →- Category
- Data engineering
- Also known as
- RTM, Real-time mode in Structured Streaming, Spark real-time mode, real-time trigger
- Verified
- 2026-10-30
Sources
- Official Official Databricks / Microsoft docs
- Official Databricks docs: Real-time mode reference (supported sources, sinks, and operators)
- Official Databricks docs: Set up real-time mode
- Official Databricks docs: Use real-time mode in Lakeflow pipelines
- Community Databricks Blog: Introducing Real-Time Mode (Public Preview, August 2025)
- Community Databricks Blog: Real-Time Mode GA (March 2026)
- Community Databricks Blog: Real-Time Mode GA in Spark Declarative Pipelines (October 2026)