Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19
Apache Spark 4.3, now available in Databricks Runtime 19, brings stateful processing to Real-Time Mode with streaming aggregation, deduplication, and custom stateful logic, hitting sub-150 ms P99 end-to-end latency across three benchmarks using familiar Structured Streaming APIs. It also adds SCD Type 2 support to Auto CDC in Python, SQL, and Spark Connect for history-preserving pipelines, plus Project Feather to cut execution overhead on eligible small-data queries and improvements to Spark Connect, Python, and SQL.
* **Bring state to Real-Time Mode:** Real-Time Mode adds streaming aggregation, deduplication, and custom stateful processing, achieving sub-150 ms P99 end-to-end latency across three representative benchmarks using familiar Structured Streaming APIs. * **Preserve history with SCD Type 2 Auto CDC:** Track how records change over time through Python, SQL, and Spark Connect, simplifying history-preserving pipelines for reporting, auditing, and historical analysis. * **Iterate faster and build with richer APIs:** Project Feather reduces execution overhead for eligible small-data queries, while improvements to Spark Connect, Python, and SQL make data and AI applications easier to develop, debug, and operate.