Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks
Summary
Databricks now supports on-demand state repartitioning for stateful Structured Streaming queries: set spark.sql.streaming.stateStore.partitions and restart on DBR 18+ with the RocksDB state store provider to redistribute state to a new partition count without rebuilding the checkpoint. This lets you resize long-running streams to match workload demands and monitor each resize through query progress metrics.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Health Plans: Your BI Tells You MLR Moved. Can Your AI Tell You Why?
Databricks and Abacus Insights combine to let payer finance teams use conversational AI to decompose MLR variances across claims, utilization, cost, and population risk in minutes instead of waiting on analyst reports. The approach embeds payer-specific business logic—covering IBNR, rebates, risk adjustment, and provider settlements—into a governed data and AI foundation so AI explains MLR shifts accurately rather than compounding confusion from inconsistent definitions across systems.
Unify your marketing data with Lakeflow Connect
Lakeflow Connect now offers native, fully managed connectors for marketing and ad platforms—including Salesforce, HubSpot, Google Ads, Meta Ads, TikTok Ads, LinkedIn Ads, Marketo, and more—landing governed data directly into Unity Catalog without any infrastructure to manage. Paired with the Ad-Genie solution accelerator, teams can turn raw ad data into a governed customer 360 and a natural-language Genie agent in just three steps.
Unifying governance across engines and catalogs in the Open Lakehouse
Apache Iceberg now has two new specs, read restrictions and catalog labels, that standardize policy enforcement across engines and catalogs by letting trusted engines handle access decisions directly and making governance metadata portable across federated catalogs. Together with centralized enforcement for untrusted engines, they give the Open Lakehouse a clear governance model covering every access pattern.
Improving Lakebase Postgres compute cache
Lakebase Postgres now runs an autoscaling cache that works alongside shared buffers to keep more data resident on compute instead of falling back to the storage layer. In production this delivered 2x throughput, fewer storage-layer reads, and lower latency compared to standard Postgres caching.
Five AI Questions We're Hearing from Financial Services Leaders
Transitioning financial services AI from pilot projects to governed production deployments hinges on addressing five key questions shaping the industry. Learn how Databricks unifies data, AI, and governance across financial crime, banking, and wealth-management workflows, and connect with leaders at Sibos 2026 in Miami.
A practical approach to end-to-end Solvency II reporting in Databricks
Databricks now supports Solvency II end to end, connecting data ingestion, reserving, capital calculation, governance, and disclosure into a single workflow instead of fragmented systems and teams. A demo shows how this unified approach delivers one control view, governed automation, AI-assisted review, and faster scenario analysis for reporting readiness.