New feature
Auto Loader
Auto Loader is a Databricks data engineering capability, introduced February 2020.
Incremental ingestion that processes new data files as they arrive in cloud storage without any additional setup, exposed as a Structured Streaming source rather than a separate service.
- You do not type its name to use it: the feature is called Auto Loader but the thing you write in code is a Structured Streaming source called cloudFiles.
- It remembers every file it has ever seen by keeping the metadata in a RocksDB key-value store inside the stream's checkpoint, which is how it promises exactly-once processing without re-listing your bucket.
- Databricks quotes its scale in units that sound like typos, describing backfills of billions of files and near real-time ingestion of millions of files per hour.
- Category
- Data engineering
- Also known as
- cloudFiles, autoloader
- Verified
- 2026-08-20
Sources
- Official Official Databricks / Microsoft docs