Photon

Photon is a Databricks compute / bi capability, introduced June 2021.

The Databricks-native vectorized query engine that accelerates SQL workloads, DataFrame API calls, ETL pipelines and stateless streaming, written to be drop-in compatible with existing Apache Spark code.

• The engine under your Spark code is not Spark - Photon is written in C++ and processes data in columnar batches rather than row by row, while still answering to the same Spark APIs. 🔗
• You can spot it working by colour in the query plan, where Photon operators and stages render peach against the blue of everything still running on Spark. 🔗
• When it meets something it cannot do it does not fail, it silently hands the work back to Spark - which is why an unsupported UDF just makes your query mysteriously ordinary again. 🔗

⚠ Limitations: No support for UDFs, RDD APIs or Dataset APIs, and only stateless streaming - stateful streaming is unsupported. Queries that already finish in under two seconds see little benefit. 🔗

Category

Compute / BI

Introduced

June 2021🔗

Also known as

Photon engine, Photon acceleration

Verified

2026-08-20