Databricks added on-demand state repartitioning for stateful Structured Streaming queries on DBR 18+.

Databricks introduced on-demand state repartitioning for stateful Structured Streaming queries running on Databricks Runtime 18 and later [1]. Operators using the RocksDB state store provider gained the ability to alter partition counts simply by updating the spark.sql.streaming.stateStore.partitions configuration and restarting their job [1]. Previously, changing partition counts forced teams to abandon existing checkpoints and rebuild state tables from raw sources. The update allowed streaming queries to scale state capacity up or down to reflect changing throughput requirements without resetting pipeline history [1].

Broader streaming workflows also received updates across the platform. Automatic change data feed reached general availability, making it easier for streaming pipelines to read row-level changes from Delta tables without manual feed management [4]. Concurrently, users shared guidance on working with Streaming Tables inside Lakeflow Spark Declarative Pipelines, focusing on pipeline behavior during continuous data processing [2].

Community forums reflected an ongoing focus on streaming operations and fundamentals. Engineers explored basic stream recovery by clarifying how checkpoint directories record read offsets and commit logs to guarantee fault tolerance [5]. Broader architecture threads focused on preferred ingestion tools for feeding streaming jobs [6], as well as the cost and operational practicality of managing long-running clusters for smaller companies [3].