Real-Time Mode
Real-Time Mode is a Databricks data engineering capability, introduced July 2025.
A trigger type for Apache Spark Structured Streaming that swaps periodic micro-batches for continuous processing, giving operational workloads such as fraud detection end-to-end latency as low as five milliseconds on the same Spark APIs.
- Its route to millisecond latency is, paradoxically, longer batches: each one runs for five minutes by default, processing data the moment it lands instead of waiting for the next micro-batch.
- The fastest streaming mode on Databricks asks you to switch off Databricks' own fast engine - the setup checklist says to turn off Photon (and autoscaling, and spot instances).
- It did not stay proprietary - the stateless half of it ships in open-source Apache Spark 4.1.
Limitations: Classic compute only on Databricks Runtime 16.4 LTS and above (dedicated access mode, or standard access mode for Python only) - serverless and Lakeflow pipelines can't run it as Structured Streaming - with autoscaling, Photon, and spot instances turned off, and enough task slots to schedule every stage at once. Update output mode only. Sources are Kafka, Event Hubs (Kafka connector), Kinesis, and AWS MSK; sinks are Kafka, Event Hubs, and forEach writers, so Delta is supported as neither. No session windows, forEachBatch, (flat)MapGroupsWithState, transformWithStateInPandas, Python UDTFs, or stream-to-stream outer joins; stream-to-table joins need a broadcast table of 10 MB or less.
Open in REbricked →- Category
- Data engineering
- Also known as
- RTM, Real-time mode in Structured Streaming, Spark real-time mode, real-time trigger
- Verified
- 2026-09-27
Sources
- Official Official Databricks / Microsoft docs
- Official Databricks docs: Real-time mode reference (supported sources, sinks, and operators)
- Official Databricks docs: Set up real-time mode
- Official Databricks docs: Use real-time mode in Lakeflow pipelines
- Community Databricks Blog: Introducing Real-Time Mode (Public Preview, August 2025)
- Community Databricks Blog: Real-Time Mode GA (March 2026)