Lakeflow pipelines

Lakeflow pipelines is the current Databricks name, previously Lakeflow Spark Declarative Pipelines (since 2026).

Delta Live Tables → Lakeflow Declarative Pipelines → Lakeflow Spark Declarative Pipelines → Lakeflow pipelines

Databricks' declarative framework for building batch and streaming data pipelines in SQL and Python, running on the Databricks Runtime.

• This is the name the docs settled on without ever announcing it - no release note declares it, the pages were simply retitled. 🔗
• It is built on Apache Spark Declarative Pipelines, an upstream open-source project, and adds the production parts on top - AUTO CDC, data quality expectations, and a queryable event log. 🔗
• The pipelines are versionless, so Databricks upgrades the runtime underneath you rather than asking you to pick a version. 🔗

⚠ Limitations: A workspace is limited to 1000 concurrent pipeline updates; a pipeline allows at most 100 source files (or 50 folder entries referencing up to 1000 files); pivot() is unsupported, Delta time travel works only on streaming tables not materialized views, and identity columns are incompatible with AUTO CDC processing. 🔗

Category

Data engineering

In use from

2026🔗

Also known as

LDP

Verified

2026-08-20