Scaling Identity Graph Ingestion to 1M Events/Sec with Spark Streaming & Delta Lake
Summary
Adobe scaled identity graph ingestion to 1 million events per second using Spark Structured Streaming and Delta Lake, a 10x improvement over their previous architecture. Key techniques included microbatching with 80% deduplication of redundant writes, repartitioning to eliminate data skew, asynchronous execution for 2x faster latency, and in-memory metadata caching using Netflix Hollow.
Summary generated by brickster.ai from the video transcript.
More from Databricks
NewsHow Jio Platforms move insights into production with Databricks | Data + AI Executive Series: Fast 5
Jio Platforms relies on Databricks Lakehouse to unify data and AI governance and achieve end to end lineage across its businesses including telecom, retail, media, and finance. The company uses Databricks Genie and Lakehouse to accelerate insights to production and allow business users to define their own KPIs and metrics.
TutorialsMigrate Databricks classic workloads to serverless with Genie Migrate | Databricks Demo
Genie Migrate is a Databricks tool designed to help users migrate workloads, including notebooks and jobs, from classic clusters to serverless. The tool automatically analyzes code and cluster infrastructure, identifies compatibility issues, and provides suggestions or applies fixes to facilitate the transition to serverless environments.
ReleasesMulti-Statement Transactions in Unity Catalog: Atomic Writes on Delta and Iceberg
Multi statement transactions in Unity Catalog allow atomic writes across multiple Delta or Iceberg tables, ensuring that updates either succeed completely or are aborted together. This capability eliminates the need for complex reconciliation logic, separate governance models, or data copying between systems.
ReleasesAutomate Data Warehouse Migration with AI - Genie Migrate for Lakehouse Intro
Genie Migrate for Lakehouse is an AI-powered migration solution that automates the movement of data and code from legacy data warehouses to the Databricks Lakehouse. It converts SQL from seven major data warehouses into open ANSI SQL and PySpark, reducing delivery timelines and ensuring high conversion accuracy.
TutorialsHow To Schedule Data Refresh in Excel | Databricks Add-in Demo
The Databricks Excel add-in allows users to connect Databricks workspaces to Excel spreadsheets, enabling data imports. Users can create scheduled refreshes to automatically update their spreadsheets with the latest data from their lakehouse.
