Apache Iceberg
Recent items mentioning Apache Iceberg across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Recent activity highlights growing interoperability and community exploration of Apache Iceberg within the Databricks ecosystem. Unity Catalog now supports catalog federation to Google Cloud's Lakehouse, enabling it to read Iceberg tables written from BigQuery and other engines 3. The community is actively exploring Spark Declarative Pipelines (SDP) with Apache Iceberg, including building OSS pipelines integrated with AWS Glue Catalog 12.
Generated daily from the 3 most recent items mentioning Apache Iceberg. Click any [N] to jump to the source.
Building a Spark Declarative Pipeline OSS with Apache Iceberg and AWS Glue Catalog
Exploring Spark Declarative Pipelines (SDP) with Apache Iceberg
I recently built a modern financial lakehouse project using Spark Declarative Pipelines (SDP), Apache Iceberg, Medallion architecture, and streaming/batch concepts. The article covers: \\- Declarative data pipelines \\- Iceberg table design \\- Bronze/Silver/Gold architecture \\- Financial analytics use cases \\- Production-style lakehouse concepts Has anyone else experimented Spark declarative pipelines in production yet ? Blog: https://medium.com/@pranavsadagopan/building-a-spark-declarative-pipeline-a-modern-financial-data-lakehouse-with-sdp-apache-iceberg-36ae6c6523ae
This release adds the dbt invocation ID to default query comments and enforces the 255-character identifier length limit for Databricks relations. It also resolves several issues, including spurious warnings for MicrobatchConcurrency and insert_overwrite, and improves handling of materialized views and Iceberg table properties.
Interoperability Between Unity Catalog and Google BigQuery via Catalog Federation
Google Cloud now supports catalog federation to Unity Catalog, enabling BigQuery users to read tables in Unity Catalog without duplication. Unity Catalog also supports catalog federation to Google Cloud's Lakehouse, allowing it to read Iceberg tables written from BigQuery and other engines.
Delta Lake 4.2.0
This release enhances Unity Catalog managed tables with support for REPLACE TABLE, RTAS, Dynamic Partition Overwrite, and improved streaming read options like `startingTimestamp` and `skipChangeCommits`. It also introduces GA support for Variant columns, Geospatial types with data skipping, and collated strings, alongside fixes for Variant stats and decimal predicates.
Delta Lake 4.1.0
Delta Lake 4.1.0 enhances Unity Catalog integration with improved support for catalog-managed tables, including atomic CTAS and conflict-free feature enablement for Deletion Vectors and Column Mapping. It also introduces a new Spark V2 connector based on Delta Kernel API for streaming reads and server-side planning capabilities.
EventsAnnouncing full Apache Iceberg™ support in Databricks
Databricks now fully supports Apache Iceberg, offering significantly higher performance for Iceberg tables compared to other vendors. This integration leverages Databricks' optimized engine and Unity Catalog for faster access and better clustering of open-format data.
NewsDatabricks: What’s new in July 2025? Updates & Features Explained! #databricks
NewsIoT for Fun & Prophet: Scaling IoT and predicting the future with Redpanda, Iceberg & Prophet
Delta Lake 3.3.0
Delta Lake 3.3.0 introduces Identity Columns, faster VACUUM LITE, and the ability to enable Row Tracking on existing tables for row-level lineage. It also allows enabling UniForm Iceberg on existing tables without data rewrites and supports reading tables with Type Widening enabled in Delta Kernel.
Events
