Apache Iceberg
Recent items mentioning Apache Iceberg across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Unity Catalog now supports multi-statement transactions that execute atomic writes across multiple Delta and Iceberg tables, removing the need for complex reconciliation logic or separate governance layers 1. Cross-platform interoperability also advanced with dbt Mesh enabling zero-copy Iceberg table sharing between Databricks and Snowflake catalogs 10, alongside new Databricks Go SDK support for Iceberg-backed machine learning data sources 2.
Generated daily from the 10 most recent items mentioning Apache Iceberg. Click any [N] to jump to the source.
The SDK introduces the Statement Execution V2 API service alongside new methods to inspect and manage security scan findings for Databricks Apps deployments. It also adds new testing parameters for pipelines, Iceberg table support for machine learning data sources, and custom container image configuration for compute environments.
Spark Declarative Pipelines - possible to write UC Managed Iceberg Tables instead of materialized views?
Hi All, Based on my experience using SDP, the tables are typically rendered as "Materialized Views". My company is making a decision to move towards Managed Iceberg for tables and I was wondering if creating Managed Iceberg tables is supported by Databricks's native SDP capabilities? submitted by /u/RazzmatazzLiving1323 [link] [comments]
Inside Apache Iceberg V3: Deletion Vectors and Puffin Files
submitted by /u/Low_Brilliant_2597 [link] [comments]
Dispelling LTAP myths
Lakebase's LTAP design makes the data lake the database's primary storage rather than an expensive replica maintained through logical replication pipelines. By pairing a single Delta or Iceberg table with Postgres caches and pageservers, point lookups read heap pages while scans read Parquet, ensuring analytical queries always see consistent snapshots that include commits not yet in the lake.
Streamhouse and Lakehouse with Apache Iceberg
submitted by /u/codingdecently [link] [comments]
Air gapped, not data dark: how Bright Machines gives customers live factory dashboards without opening the plant
Bright Machines replaced a fragile nine-service pipeline by using Databricks Unity Catalog federation with AWS Glue to query edge-captured Apache Iceberg data from air-gapped plants in place. This unified architecture makes factory PostgreSQL changes queryable within two minutes, powering 10-minute AI/BI Dashboard refreshes and zero-copy sharing under a single governed platform.
Catalog Linked Database - Snowflake writes UC Managed Iceberg tables to Unity Catalog
Is Unity Catalog REST API only accessible via public internet from Snowflake when attempting to set-up a private link connection (catalog integration) to a Unity Catalog (Catalog Linked Database)? I need a private link connection between Databricks and Snowflake. The use case is Snowflake needs to be able to write Managed Iceberg tables to Unity Catalog. submitted by /u/RazzmatazzLiving1323 [link] [comments]
Apache Iceberg on Snowflake got a lot simpler in 2026, and dbt's new catalog config is what makes it click
Apache Iceberg on Snowflake got much simpler in 2026, with dbt's new catalog config doing the key work of tying the setup together. Together with changes in Snowflake and DuckDB, it's now simple enough for a small team to get running in an afternoon.
What Open Data Infrastructure looks like with dbt Mesh
A new catalogs.yml v2 spec decouples dbt's Iceberg configuration from Snowflake-specific catalog integrations, and a live dbt Mesh demo showed three teams on Snowflake and Databricks sharing one set of Iceberg tables through Horizon and Unity catalogs via catalog federation, with no data copied. The session also introduced Managed Data Lake, which writes Iceberg or Delta metadata to a customer's own cloud storage using the open source Polaris catalog, and Lake Compute (private beta), which routes individual dbt models to DuckDB while the rest of the project stays on Snowflake.
How Obie Insurance cut its month-end close from 25 hours to 2 with dbt State and a lake house rebuild
Obie Insurance rebuilt its data platform on dbt and Apache Iceberg in a lake house architecture, cutting a critical month-end close process from over 25 hours to 2 hours. The post walks through how dbt State and the lake house rebuild made that reduction possible.
Apache Iceberg TCO: The Real Cost of Lakehouse Table Maintenance
submitted by /u/codingdecently [link] [comments]
What makes a catalog open? Unity Catalog Advances the Open Lakehouse
Unity Catalog is the first catalog to support the Apache Iceberg REST catalog spec for REGISTER and UNREGISTER, enabling seamless table movement across catalogs without data duplication. By keeping managed tables in customer-owned storage, it provides full data portability while delivering advanced catalog features directly on tables you own.
Real-Time Lakehouse Guide: Apache Iceberg in Production
submitted by /u/codingdecently [link] [comments]
Automatic CDF
The recipe is simple: combine two great features of Delta, time travel and row tracking ID, and you will get Automatic Change Data Feed. As Elizabeth Bowman, databricks PM said: "Teams should be able to use table changes without asking an administrator to enable a separate feed. Automatic CDF makes this possible on supported Delta and managed Iceberg tables while reducing the write overhead of legacy CDF." https://medium.com/databrickscommunity/automatic-change-data-feed-cdf-is-here-to-save-space-improve-write-time-and-feed-change-data-076131a6abb0 https://www.sunnydata.ai/blog/automatic-cdf-databricks submitted by /u/hubert-dudek [link] [comments]
Transactions are Generally Available, advancing warehousing on the Lakehouse
Multi-statement transactions are now generally available for Delta Lake and Apache Iceberg tables on Databricks, bringing native atomic guarantees to multi-table workflows. Backed by row-level concurrency and Unity Catalog governance, this capability allows practitioners to run critical warehouse workloads directly on open lakehouse data without maintaining separate warehouse systems.
UnityCatalog 0.7.0
Unity Catalog 0.7.0 enables native Apache Iceberg REST catalog support by default, allowing external query engines to read, write, and manage Iceberg tables directly. Dropping managed tables, volumes, or models now automatically and permanently purges underlying cloud storage files after a default seven-day delay.
Announcing Workday Data Connect federation in Unity Catalog
The new Workday Data Connect connector (Beta) brings zero-copy federation to Unity Catalog, allowing teams to query Workday's shared Iceberg tables directly from cloud storage without ingestion pipelines or data duplication. Queries run on Databricks compute under Unity Catalog governance, enabling you to combine live HR and financial data with existing Databricks datasets for Genie-powered exploration, workforce analytics, and AI.
Delta Lake 4.4.1
Delta Lake 4.4.1 adds safeguards for catalog-managed tables by blocking unsupported cleanups and skipping automatic maintenance so writes can succeed reliably. It also fixes snapshot construction errors caused by unratified staged commits, allowing safe re-enablement of S3A fast listing.
Read Restrictions and Catalog Labels: Unifying governance across engines and catalogs
Apache Iceberg has introduced read restrictions and catalog labels, two new specifications designed to standardize policy enforcement across different engines and catalogs. These additions eliminate fragmented enforcement by delegating access decisions to trusted engines and making governance metadata portable across federated catalogs.
Writing to externally managed Iceberg tables from Databricks: am I missing something?
I recently joined a new company as a data architect and I'm trying to understand how things work here today (new to Databricks, main experience with RedShift and BigQuery), and I'd like corrections from people who use Databricks daily. I've been testing how Databricks handles Iceberg tables that are managed by Glue. What I'm seeing: Can read tables as Databricks' federation lets me mount the external catalog and query the tables as foreign tables. Can't write (is this accurate?) INSERT, MERGE, and UPDATE operations against those foreign Iceberg tables fail, and the Databricks docs seem to confirm this. Question: Has anyone gotten writes to non-UC Iceberg tables working from Databricks? That could be through a config, a preview feature, or an open-source Spark Iceberg runtime with a REST catalog on a cluster. If I've got any of this wrong, please correct me as I am still getting up to speed with the platform. submitted by /u/DonutSignificant4174 [link] [comments]
The Story of iceberg-rust: The Official Rust Implementation of Apache Iceberg
The story of iceberg-rust is a great example of how open-source communities work and how new projects are created and developed. That’s the spirit of the community-over-code philosophy. submitted by /u/Low_Brilliant_2597 [link] [comments]
How AI Agents Query Apache Iceberg Data with MCP
submitted by /u/codingdecently [link] [comments]
Unity Catalog: Pros and Cons
Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began. In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems. That is where Unity Catalog comes in. submitted by /u/Low_Brilliant_2597 [link] [comments]
Unifying governance across engines and catalogs in the Open Lakehouse
Apache Iceberg now has two new specs, read restrictions and catalog labels, that standardize policy enforcement across engines and catalogs by letting trusted engines handle access decisions directly and making governance metadata portable across federated catalogs. Together with centralized enforcement for untrusted engines, they give the Open Lakehouse a clear governance model covering every access pattern.
Apache Iceberg Table Cleanup: A Production Guide
A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle. submitted by /u/codingdecently [link] [comments]
Apache Iceberg Compaction Best Practices
submitted by /u/codingdecently [link] [comments]
Automating Apache Iceberg Table Maintenance
submitted by /u/codingdecently [link] [comments]
Apache Iceberg Performance Optimization: Queries to Tables
submitted by /u/codingdecently [link] [comments]
Data Lakehouse with Apache Iceberg: A Guide
submitted by /u/codingdecently [link] [comments]
Open Table Formats Explained: Iceberg vs. Delta vs. Hudi
Delta Lake and Apache Iceberg now share governance through catalog-coordinated commits, while metadata trees and transaction logs enable data skipping and time travel that cut query costs without sacrificing access to prior table versions. Interoperability features like Delta Lake UniForm and Unity Catalog let teams query the same underlying data natively as either format across engines like Spark and Trino, reducing vendor lock-in.
Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files
submitted by /u/codingdecently [link] [comments]
Got error using Opensharing Get Table API to foreign iceberg table
UnityCatalog 0.6.0
This release adds Apache Spark 4.2 support and introduces metric views and SQL views as governed catalog objects for semantic layers and query management. UC access tokens now expire after 24 hours by default, requiring periodic re-exchange instead of the previous indefinite validity.
Delta Lake 4.4.0
Delta Lake 4.4.0 supports Apache Spark 4.2 and adds identity columns and generated-column support in SQL DDL. The release expands UC Delta API integration to Delta Kernel and the experimental Flink connector for catalog-managed tables, introduces Flink upsert mode with merge-on-read updates, and improves VOID-column and partition handling.
[Announce] Apache Iceberg Virtual Meetup Series
submitted by /u/PrideDense2206 [link] [comments]
Delta Lake 3.3.3
Delta 3.3.3 fixes transaction log retention bugs that broke time travel and CDF reads, and a Delta Sharing deletion vector cache bug causing long-running queries to fail. It adds opt-in RANDOMIZE_FILE_PREFIXES to spread S3 object keys for high-throughput workloads and upgrades Delta Sharing client for improved OAuth and retry handling.
dbt Summit 2026: the keynotes and product sessions
dbt Summit 2026 previews major platform evolutions, including a Rust-based dbt Core rebuild with Arrow adapters, Parquet-backed artifacts, and dbt State to eliminate redundant model runs and reduce compute spend by 30%. The upcoming sessions also focus on open lakehouse architectures with Apache Iceberg, portable semantic standards via Apache Ossie, and governed agentic workflows using dbt Wizard.
oreign Iceberg tables via HMS federation fail with browse_only_table on all compute — request Public
Delta Lake 4.3.1
Delta Lake 4.3.1 fixes OAuth authentication failures in the Delta REST Catalog caused by incorrect key lowercasing and enables S3A fast listing when using OSS UnityCatalog's CredScopedFileSystem wrapper. It also prevents the reserved is_managed_location property from persisting into managed table metadata.
Delta Lake 4.3.0
Delta 4.3.0 deepens Unity Catalog integration by making it the source of truth for managed table operations via the UC Delta REST API, introduces replaceOn/replaceUsing DataFrame APIs for selective row-level data replacement, and improves UniForm with atomic Iceberg conversion and incremental metadata updates. Delta Sharing gains streaming support, Change Data Feed capabilities, and Trigger.AvailableNow, plus performance improvements like better V2 checkpoint parallelization and variant column statistics for data skipping.
Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
Databricks has introduced OpenSharing, a Linux Foundation-hosted evolution of Delta Sharing that expands zero-copy sharing beyond tabular data to models, agents, and semantic context across Delta Lake, Apache Iceberg, and Parquet. The enterprise implementation on Databricks enables governed sharing of Genie Agents via Unity Catalog, automated private networking with SecureConnect, and low-latency multi-cloud access via Global Distribution.
Unity Catalog and the next era of Apache Iceberg
Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance
Unity Catalog now offers GA support for Managed Iceberg, Iceberg v3, and Foreign Iceberg, making it the most comprehensive and production-ready Apache Iceberg catalog with open APIs, catalog federation, and secure sharing. Future versions of Iceberg and Delta will converge on a unified metadata structure, eliminating the tradeoff between interoperability and performance.
Interoperability Between Unity Catalog and Google BigQuery via Catalog Federation
Google Cloud now supports catalog federation to Unity Catalog, enabling BigQuery users to read tables in Unity Catalog without duplication. Unity Catalog also supports catalog federation to Google Cloud's Lakehouse, allowing it to read Iceberg tables written from BigQuery and other engines.
Delta Lake 4.2.0
Delta 4.2.0 enables atomic REPLACE TABLE, RTAS, and DPO for catalog-managed tables, enhances streaming capabilities, and adds a Kernel-based Flink connector. The release makes Variant generally available, adds geospatial and collation support, and includes comprehensive security hardening.
Delta Lake 4.1.0
Delta Lake 4.1.0 supports Apache Spark 4.1.0 and introduces conflict-free enablement of Deletion Vectors and Column Mapping on existing tables without blocking concurrent writes. The release requires Java 17 and Spark 4.0.1 or higher (dropping Spark 3.5), adds full catalog-managed table support in Delta Kernel for Unity Catalog integration, and fixes MERGE/INSERT struct expansion bugs.
Delta Lake 3.3.0
Delta Lake 3.3.0 adds Identity Columns for automatic unique keys, VACUUM LITE for faster transaction log-based cleanup, and enables Row Tracking backfill on existing tables for row-level lineage tracking. UniForm Iceberg can now be enabled on existing Delta tables without data rewriting, and Type Widening is now supported in Delta Kernel for reading type-evolved tables.
Get Tuesday's version of this
Tracking Apache Iceberg? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.