Skip to content
All videos
newsDatabricks·August 30, 2021

How Databricks Leverages Auto Loader to Ingest Millions of Files an Hour

Summary

Auto Loader, a Structured Streaming source in Databricks, handles continuous, incremental ingestion from cloud storage with efficient file discovery through notifications or incremental listing and exactly-once guarantees across billions of files. The talk covers schema inference, drift detection and evolution, and how Databricks uses Auto Loader internally to ingest millions of files every hour.

Summary generated by brickster.ai from the video transcript.

More from Databricks