/
New feature GA

Real-Time Mode

Real-Time Mode is a Databricks data engineering capability, introduced July 2025.

A trigger type for Apache Spark Structured Streaming that swaps periodic micro-batches for long-running batches processed as data arrives, up to 100x faster than micro-batch mode, for operational workloads such as fraud detection on the same Spark APIs.

  • Its route to millisecond latency is, paradoxically, longer batches: each one runs for five minutes by default, processing data the moment it lands instead of waiting for the next micro-batch.
  • The fastest streaming mode on Databricks asks you to switch off Databricks' own fast engine - the setup checklist says to turn off Photon (and autoscaling, and spot instances).
  • It did not stay proprietary - the stateless half of it ships in open-source Apache Spark 4.1.

Limitations: Direct Structured Streaming queries need classic compute on Databricks Runtime 16.4 LTS and above (dedicated access mode, or standard access mode for Python only) with autoscaling, Photon, and spot instances turned off; serverless notebooks and jobs can't run them, so serverless real-time processing goes through Lakeflow pipelines instead. Update output mode only. Sources are Kafka, Event Hubs (Kafka connector), Kinesis, and AWS MSK; Delta is supported as neither a source nor a sink. Stream-to-stream outer joins aren't supported, and stream-to-table joins need a broadcast table of 10 MB or less.

Open in REbricked →
Category
Data engineering
Introduced
July 2025
Announced at
GA announced March 19, 2026
Also known as
RTM, Real-time mode in Structured Streaming, Spark real-time mode, real-time trigger
Verified
2026-10-30

Sources

Related in Data engineering