Auto Loader
Auto Loader is a Databricks data engineering capability, introduced February 2020.
Incremental ingestion that processes new data files as they arrive in cloud storage without any additional setup, exposed as a Structured Streaming source rather than a separate service.
- You do not type its name to use it: the feature is called Auto Loader but the thing you write in code is a Structured Streaming source called cloudFiles.
- It remembers every file it has ever seen by keeping the metadata in a RocksDB key-value store inside the stream's checkpoint, which is how it promises exactly-once processing without re-listing your bucket.
- Databricks quotes its scale in units that sound like typos, describing backfills of billions of files and near real-time ingestion of millions of files per hour.
Limitations: Auto Loader does not guarantee the order in which files are discovered or processed, so pipelines must handle out-of-order arrivals. With file events, the cache can expire if the stream runs infrequently (invoke it at least once every seven days or it falls back to directory listing), and useIncremental, useNotifications, cloudFiles.fetchParallelism, cloudFiles.backfillInterval and cloudFiles.pathRewrites are unsupported. Classic file notification mode does not support changing the source path.
Open in REbricked →- Category
- Data engineering
- Also known as
- cloudFiles, autoloader
- Verified
- 2026-09-20
Sources
- Official Official Databricks / Microsoft docs