Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19
Summary
Apache Spark 4.3, now available in Databricks Runtime 19, brings stateful processing to Real-Time Mode with streaming aggregation, deduplication, and custom stateful logic, hitting sub-150 ms P99 end-to-end latency across three benchmarks using familiar Structured Streaming APIs. It also adds SCD Type 2 support to Auto CDC in Python, SQL, and Spark Connect for history-preserving pipelines, plus Project Feather to cut execution overhead on eligible small-data queries and improvements to Spark Connect, Python, and SQL.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
The Enterprise Case for Open-Weight Models on Databricks
Open-weight models now approach frontier-class intelligence at lower cost, and the post argues this gives enterprises data sovereignty and operational autonomy without a large capability tradeoff. Unity AI Gateway serves open-weight, frontier, and external models through one governed API, with fast inference, smart routing, fallbacks, and budget controls.
Declare and optimize with UNIQUE constraints on Databricks
Databricks UNIQUE constraints let you declare multiple nullable or composite uniqueness guarantees beyond a table's single PRIMARY KEY and reference them from foreign keys. Once you've validated and continuously maintained these informational constraints, marking them RELY lets the optimizer eliminate redundant joins and DISTINCT aggregations, removing over 99.99% of shuffle and speeding up queries by up to 12× in 10 TB TPC-DS benchmarks.
Claude Platform now supports native OpenTelemetry observability in Databricks MLflow
Claude Platform's native OpenTelemetry export is now in Public Beta, sending traces from Anthropic SDKs, the Messages API, and Claude Managed Agents into Databricks MLflow, where they're stored in Unity Catalog. Use trace analysis and MLflow Issues to investigate bad answers and spot repeated tool calls, then have domain experts review answers and rerun evaluation on the same test cases to check whether changes improve quality and cut unnecessary work.
Lakebase Postgres is now generally available on Google Cloud
Lakebase Postgres, Databricks' fully managed serverless Postgres database, is now generally available on Google Cloud, with autoscaling, scale to zero, instant branching, and point-in-time recovery built in. Because operational and analytical data share the same storage foundation, analytics, apps, and agents run on one platform without fragile pipelines to maintain.
Lakebase branching for CI/CD: lessons from 100 PR databases
Snorkel AI's CI/CD pipeline now gives every open pull request its own Lakebase Postgres branch, a full read/write copy of production data that's ready in under a minute rather than the 30+ minutes that previously made per-PR copies impractical. Because branches are copy-on-write, 100 concurrent PR databases on a ~2.5 TB production dataset add storage only for what each one changes, with TTL expiration cleaning up automatically and private connectivity plus Okta SSO keeping the setup governed.
Databricks and SAP unleash end-to-end data & AI capabilities to level up enterprise AI transformation
Databricks and SAP are expanding their partnership to give customers broader access to SAP application data across cloud and on-premises, speeding time to value for analytics and AI. Genie, Lakeflow, Unity Gateway, and Lakebase now come to SAP Databricks fully integrated with SAP Business Data Cloud, so teams can use the full Data + AI Platform on all their SAP data.