Python asynchronous TransformWithState and on-demand state repartitioning reached general availability

Two stateful streaming features reached general availability in October 2026. Streaming On-Demand State Repartitioning was announced as GA on October 13 [11], and Python Asynchronous TransformWithState for streaming followed on October 23 [5]. Both announcements were short community posts, so this entry does not describe their behavior beyond what the titles state.

On the pipelines side, Databricks described writing to one managed Unity Catalog table from several sources, with one flow per source. The same item said existing Structured Streaming jobs could be migrated without reprocessing source data by importing their checkpoints [3].

Real-Time Mode was the main theme in the other announcements. A news item and a video described energy trading on Databricks, with Spark Structured Streaming Real-Time Mode feeding market data into Lakebase, a serverless Postgres service. That data supported a trading app, ML models and dashboards, with Unity Catalog providing governance [4][9]. A separate video built a digital twin of a manufacturing cell on Real-Time Mode, Lakebase, Databricks Apps and Unity Catalog. It used sensor readings processed as they arrive to catch faults such as a hot spindle [1]. A community thread also discussed Real-Time Mode in the context of a SaaS product [6].

Community questions were mostly operational. One user asked for alternatives to dropMalformed and reported that a Schema Registry URL was not supported for the Kafka source on a Shared cluster [2]. Another described a streaming job on a job cluster ending with "driver unresponsive" after 8 to 12 hours despite a healthy heap and plenty of memory [7]. A third raised inefficiencies in writeStream when one stream feeds N sinks [8].