Summary
Spark Declarative Pipelines (SDP) are simplifying complex ETL and streaming workloads, pioneering the next generation of data engineering. Get a deep dive into Enzyme, our incremental view maintenance engine, which won an honorable mention at SIGMOD.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Enabling Evolutionary Database Development: database branching with Lakebase
Why this series existsThe methodology described in Evolutionary Database Design and...
Winning under CMS TEAM: Building the learning health system to realize success in VBC today and tomorrow
Databricks helps healthcare providers succeed under the mandatory CMS TEAM program by building an AI-enabled data foundation for proactive, data-driven intervention. This enables a unified view across clinical and claims data, embedding predictive insights into care workflows to reduce SNF costs by 15% and readmissions by 12%.
How enterprise leaders are scaling AI agents across their organization
Databricks practitioners can learn five key practices for scaling agentic AI responsibly across enterprise core workflows like HR, finance, and fraud detection. This post helps leaders deliver rapid gains from AI agents while maintaining governance, trust, and cost control.
Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance
Unity Catalog now offers GA support for Managed Iceberg, Iceberg v3, and Foreign Iceberg, making it the most comprehensive and production-ready Apache Iceberg catalog with open APIs, catalog federation, and secure sharing. Future versions of Iceberg and Delta will converge on a unified metadata structure, eliminating the tradeoff between interoperability and performance.
Reliable LLM Inference at Scale
Databricks now offers model units, a VM-like abstraction for allocating and scaling GPU resources per customer, enabling cost-aware load balancing and autoscaling that saved over 80% in GPU costs. Runtime reliability mechanisms like black-box health checks and multimodal bottleneck profiling further improve throughput and recover from silent failures automatically.
BI Serving Pointers; Maximizing for Performance and TCO
Databricks now offers Unity Catalog Metric Views for a headless semantic layer, enabling governed business metrics across all BI tools and AI agents. Maximize performance and TCO by structuring your physical layer with star schemas, liquid clustering, and Predictive Optimization, and leverage aggregate-aware materialization for OLAP-style performance.