Apache Iceberg
Recent items mentioning Apache Iceberg across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Apache Iceberg introduced two new specifications—read restrictions and catalog labels—standardizing policy enforcement across engines and making governance metadata portable across federated catalogs 2. Within the Databricks ecosystem, catalog-coordinated commits and Delta Lake UniForm now provide shared governance and native cross-format querying between Iceberg and Delta Lake 8, amid an influx of operational guidance focused on automating production table maintenance, compaction, and cleanup 345.
Generated daily from the 10 most recent items mentioning Apache Iceberg. Click any [N] to jump to the source.
Unity Catalog: Pros and Cons
Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began. In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems. That is where Unity Catalog comes in. submitted by /u/Low_Brilliant_2597 [link] [comments]
Unifying governance across engines and catalogs in the Open Lakehouse
Apache Iceberg now has two new specs, read restrictions and catalog labels, that standardize policy enforcement across engines and catalogs by letting trusted engines handle access decisions directly and making governance metadata portable across federated catalogs. Together with centralized enforcement for untrusted engines, they give the Open Lakehouse a clear governance model covering every access pattern.
Apache Iceberg Table Cleanup: A Production Guide
A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle. submitted by /u/codingdecently [link] [comments]
Apache Iceberg Compaction Best Practices
submitted by /u/codingdecently [link] [comments]
Automating Apache Iceberg Table Maintenance
submitted by /u/codingdecently [link] [comments]
Apache Iceberg Performance Optimization: Queries to Tables
submitted by /u/codingdecently [link] [comments]
Data Lakehouse with Apache Iceberg: A Guide
submitted by /u/codingdecently [link] [comments]
Open Table Formats Explained: Iceberg vs. Delta vs. Hudi
Delta Lake and Apache Iceberg now share governance through catalog-coordinated commits, while metadata trees and transaction logs enable data skipping and time travel that cut query costs without sacrificing access to prior table versions. Interoperability features like Delta Lake UniForm and Unity Catalog let teams query the same underlying data natively as either format across engines like Spark and Trino, reducing vendor lock-in.
Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files
submitted by /u/codingdecently [link] [comments]
Got error using Opensharing Get Table API to foreign iceberg table
UnityCatalog 0.6.0
This release adds Apache Spark 4.2 support and introduces metric views and SQL views as governed catalog objects for semantic layers and query management. UC access tokens now expire after 24 hours by default, requiring periodic re-exchange instead of the previous indefinite validity.
Delta Lake 4.4.0
Delta Lake 4.4.0 supports Apache Spark 4.2 and adds identity columns and generated-column support in SQL DDL. The release expands UC Delta API integration to Delta Kernel and the experimental Flink connector for catalog-managed tables, introduces Flink upsert mode with merge-on-read updates, and improves VOID-column and partition handling.
[Announce] Apache Iceberg Virtual Meetup Series
submitted by /u/PrideDense2206 [link] [comments]
Delta Lake 3.3.3
Delta 3.3.3 fixes transaction log retention bugs that broke time travel and CDF reads, and a Delta Sharing deletion vector cache bug causing long-running queries to fail. It adds opt-in RANDOMIZE_FILE_PREFIXES to spread S3 object keys for high-throughput workloads and upgrades Delta Sharing client for improved OAuth and retry handling.
oreign Iceberg tables via HMS federation fail with browse_only_table on all compute — request Public
Delta Lake 4.3.1
Delta Lake 4.3.1 fixes OAuth authentication failures in the Delta REST Catalog caused by incorrect key lowercasing and enables S3A fast listing when using OSS UnityCatalog's CredScopedFileSystem wrapper. It also prevents the reserved is_managed_location property from persisting into managed table metadata.
ReleasesLakehouse//RT, the real-time Lakehouse powered by Reyden — Reynold Xin, Co–founder & Chief Architect
Databricks introduces Lakehouse//RT, a new SQL warehouse powered by the Raiden engine, designed to provide millisecond performance and massive concurrency for real-time analytics directly on data lake formats like Delta and Iceberg. This innovation aims to unify data warehousing and serving stacks, eliminating the need for separate systems and data copies.
EventsNo one needs to care about table formats with Databricks' Ryan Blue, creator of Apache Iceberg
Databricks announced the GA release of Iceberg v3, which unifies data layers so files can be shared across Delta and Iceberg tables without rewriting. The company is also working towards a unified metadata layer in Delta 5 and Iceberg v4, aiming for a full unification vision later this year.
Delta Lake 4.3.0
Delta 4.3.0 deepens Unity Catalog integration by making it the source of truth for managed table operations via the UC Delta REST API, introduces replaceOn/replaceUsing DataFrame APIs for selective row-level data replacement, and improves UniForm with atomic Iceberg conversion and incremental metadata updates. Delta Sharing gains streaming support, Change Data Feed capabilities, and Trigger.AvailableNow, plus performance improvements like better V2 checkpoint parallelization and variant column statistics for data skipping.
Unity Catalog and the next era of Apache Iceberg
Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance
Unity Catalog now offers GA support for Managed Iceberg, Iceberg v3, and Foreign Iceberg, making it the most comprehensive and production-ready Apache Iceberg catalog with open APIs, catalog federation, and secure sharing. Future versions of Iceberg and Delta will converge on a unified metadata structure, eliminating the tradeoff between interoperability and performance.
NewsApache Iceberg V3 on Databricks: From Ingestion to Analytics
The video demonstrates Apache Iceberg v3 on Databricks, showcasing how its new variant column type natively handles semi-structured data and how row-level concurrency enables simultaneous data ingestion and corrections. It also highlights cross-platform data accessibility from open-source Spark via the Iceberg REST catalog, ensuring no vendor lock-in.
Building a Spark Declarative Pipeline OSS with Apache Iceberg and AWS Glue Catalog
Exploring Spark Declarative Pipelines (SDP) with Apache Iceberg
I recently built a modern financial lakehouse project using Spark Declarative Pipelines (SDP), Apache Iceberg, Medallion architecture, and streaming/batch concepts. The article covers: \\- Declarative data pipelines \\- Iceberg table design \\- Bronze/Silver/Gold architecture \\- Financial analytics use cases \\- Production-style lakehouse concepts Has anyone else experimented Spark declarative pipelines in production yet ? Blog: https://medium.com/@pranavsadagopan/building-a-spark-declarative-pipeline-a-modern-financial-data-lakehouse-with-sdp-apache-iceberg-36ae6c6523ae
Interoperability Between Unity Catalog and Google BigQuery via Catalog Federation
Google Cloud now supports catalog federation to Unity Catalog, enabling BigQuery users to read tables in Unity Catalog without duplication. Unity Catalog also supports catalog federation to Google Cloud's Lakehouse, allowing it to read Iceberg tables written from BigQuery and other engines.
The Next Era of the Open Lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
This is a great news for Apache Iceberg users on Databricks. V3 is bringing some interesting features from Delta to Apache Iceberg.
Delta Lake 4.2.0
Delta 4.2.0 enables atomic REPLACE TABLE, RTAS, and DPO for catalog-managed tables, enhances streaming capabilities, and adds a Kernel-based Flink connector. The release makes Variant generally available, adds geospatial and collation support, and includes comprehensive security hardening.
Delta Lake 4.1.0
Delta Lake 4.1.0 supports Apache Spark 4.1.0 and introduces conflict-free enablement of Deletion Vectors and Column Mapping on existing tables without blocking concurrent writes. The release requires Java 17 and Spark 4.0.1 or higher (dropping Spark 3.5), adds full catalog-managed table support in Delta Kernel for Unity Catalog integration, and fixes MERGE/INSERT struct expansion bugs.
EventsAnnouncing full Apache Iceberg™ support in Databricks
Databricks now fully supports Apache Iceberg, offering significantly higher performance for Iceberg tables compared to other vendors. This integration leverages Databricks' optimized engine and Unity Catalog for faster access and better clustering of open-format data.
NewsDatabricks: What’s new in July 2025? Updates & Features Explained! #databricks
Databricks released a free serverless tier and introduced PostgreSQL support for real-time transactional queries alongside major feature updates including AI document parsing, redesigned declarative pipelines, tag-based data masking policies, and GPU acceleration. Runtime 17 adds merge-into syntax for PySpark, SQL procedures in Unity Catalog, recursive CTEs, and automatic data classification for GDPR compliance.
NewsNo More Fragile Pipelines: Kafka and Iceberg the Declarative Way
Kafka topics and Iceberg tables have incompatible data models that cause problems with schema evolution, consistency, and file fragmentation when building data pipelines. Confluent's Table Flow solution declaratively handles schema evolution, manages consistency through two-phase commits with segments, and performs automatic compaction to optimize query performance.
NewsIoT for Fun & Prophet: Scaling IoT and predicting the future with Redpanda, Iceberg & Prophet
An IoT pipeline captures temperature and humidity data from an ESP32 device via AWS IoT, streams it through Red Panda into Iceberg tables for querying. Prophet time series modeling then forecasts future environmental conditions based on the collected data.
Delta Lake 3.3.0
Delta Lake 3.3.0 adds Identity Columns for automatic unique keys, VACUUM LITE for faster transaction log-based cleanup, and enables Row Tracking backfill on existing tables for row-level lineage tracking. UniForm Iceberg can now be enabled on existing Delta tables without data rewriting, and Type Widening is now supported in Delta Kernel for reading type-evolved tables.
EventsThe Future of Lakehouse Format Interoperability with Ali Ghodsi and Ryan Blue at Data + AI Summit
Get Tuesday's version of this
Tracking Apache Iceberg? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.