New feature
Pandas API on Spark
Pandas API on Spark is a Databricks data engineering capability, introduced October 2021.
Koalas Pandas API on Spark
The pandas-equivalent DataFrame API built into Apache Spark itself (pyspark.pandas), letting pandas users work on Spark-scale data and PySpark users do pandas-only tricks such as plotting straight from a DataFrame.
- It is not a Databricks library at all - it lives inside Apache Spark since 3.2, so on Databricks Runtime 10.0 and above it is one import away as pyspark.pandas, no install required.
- Its headline party trick for PySpark users is plotting data directly from a PySpark DataFrame, a task the docs politely describe as difficult to do with PySpark alone.
- Databricks Runtime 10.0, the release that carried it onto the platform, went GA on October 20, 2021 in the same breath as the Photon public preview.
- Category
- Data engineering
- Also known as
- pandas API on Spark, pyspark.pandas, pandas-on-Spark
- Verified
- 2026-09-11