All videos
newsDatabricks·July 7, 2025
Scaling Identity Graph Ingestion to 1M Events/Sec with Spark Streaming & Delta Lake
Summary
Adobe scaled identity graph ingestion to 1 million events per second using Spark Structured Streaming and Delta Lake, a 10x improvement over their previous architecture. Key techniques included microbatching with 80% deduplication of redundant writes, repartitioning to eliminate data skew, asynchronous execution for 2x faster latency, and in-memory metadata caching using Netflix Hollow.
Summary generated by brickster.ai from the video transcript.
More from Databricks
NewsTeach AI how your business actually runs
Model intelligence is no longer the bottleneck for enterprise AI adoption because modern frontier models easily handle complex reasoning tasks. Business value requires providing these models with specific organizational context and metadata about internal processes to create a competitive advantage.
Databricks3d ago