Skip to content
All videos
newsDatabricks·July 28, 2021

How Adobe uses Structured Streaming at Scale

Summary

Adobe shares Structured Streaming lessons from its Unified Profile System, which ingests TBs of data a day and is PBs large. Topics include monitoring Kafka consumer lag, backpressure, minPartitions and maxOffsetsPerTrigger tuning, micro-batching practices, windowing, the RocksDB state store, broadcast joins, custom aggregators and off-heap counters in Redis.

Summary generated by brickster.ai from the video transcript.

More from Databricks