All videos
newsDatabricks·July 26, 2023
Unlocking Near Real Time Data Replication with CDC, Apache Spark™ Streaming, and Delta Lake
Summary
DoorDash implemented an open-source near real-time data ingestion framework called Pepto using Apache Kafka, Apache Spark Streaming, and Delta Lake. The system captures change data feeds from various transactional databases and performs micro-batched upserts to achieve a P99 data latency of 7 to 30 minutes.
Summary generated by brickster.ai from the video transcript.
More from Databricks
NewsTeach AI how your business actually runs
Model intelligence is no longer the bottleneck for enterprise AI adoption because modern frontier models easily handle complex reasoning tasks. Business value requires providing these models with specific organizational context and metadata about internal processes to create a competitive advantage.
Databricks3d ago