Skip to content
All videos
newsDatabricks·July 26, 2023

Structured Streaming: Demystifying Arbitrary Stateful Operations

Summary

This video demonstrates how to implement arbitrary stateful operations in Spark Structured Streaming using flatMapGroupsWithState in Scala and applyInPandasWithState in Python. It breaks down the implementation into four core components: data structures, initialization logic, steady-state logic, and timeout logic.

Summary generated by brickster.ai from the video transcript.

More from Databricks