Delta Lake
Recent items mentioning Delta Lake across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Delta Lake 3.3.3 patches transaction log retention bugs that were breaking time travel and CDF reads, plus a Delta Sharing deletion vector cache bug causing long-running query failures 2. It also adds opt-in RANDOMIZE_FILE_PREFIXES for spreading S3 object keys under high-throughput writes and upgrades the Delta Sharing client's OAuth and retry handling 2. Separately, Databricks' new beta FILE column type lets Delta tables natively store unstructured data like documents, images, audio, and video under unified governance 3.
Generated daily from the 4 most recent items mentioning Delta Lake. Click any [N] to jump to the source.
🇪🇸 ☕ ¿Cómo organiza Databricks tus datos? | De Unity Catalog a Delta Lake ⚡
Delta Lake 3.3.3
Delta 3.3.3 fixes transaction log retention bugs that broke time travel and CDF reads, and a Delta Sharing deletion vector cache bug causing long-running queries to fail. It adds opt-in RANDOMIZE_FILE_PREFIXES to spread S3 object keys for high-throughput workloads and upgrades Delta Sharing client for improved OAuth and retry handling.
Introducing FILE type: a native column type for multimodal data
The new FILE column type, now in beta, allows you to store unstructured data like documents, images, audio, and video natively inside your tables. This enables unified governance with fine-grained access controls
How Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale
CFL) on the
Delta Lake 4.3.1
Delta Lake 4.3.1 fixes OAuth authentication failures in the Delta REST Catalog caused by incorrect key lowercasing and enables S3A fast listing when using OSS UnityCatalog's CredScopedFileSystem wrapper. It also prevents the reserved is_managed_location property from persisting into managed table metadata.
python-v1.6.1: Column Mapping write support
This release adds column mapping write support and BlindDeltaTable for stats-free appends, alongside improvements to data skipping and partition pruning. S3DynamoDbLogStore has been removed in preparation for 1.0.0.
ReleasesLakehouse//RT, the real-time Lakehouse powered by Reyden — Reynold Xin, Co–founder & Chief Architect
Databricks introduces Lakehouse//RT, a new SQL warehouse powered by the Raiden engine, designed to provide millisecond performance and massive concurrency for real-time analytics directly on data lake formats like Delta and Iceberg. This innovation aims to unify data warehousing and serving stacks, eliminating the need for separate systems and data copies.
EventsNo one needs to care about table formats with Databricks' Ryan Blue, creator of Apache Iceberg
Databricks announced the GA release of Iceberg v3, which unifies data layers so files can be shared across Delta and Iceberg tables without rewriting. The company is also working towards a unified metadata layer in Delta 5 and Iceberg v4, aiming for a full unification vision later this year.
Data Lake vs. Cloud Data Warehouse: A Practical Guide for Data Scientists
Data lakes offer schema-on-read flexibility for ML and advanced analytics, while cloud data warehouses prioritize schema-on-write for high-concurrency BI. Lakehouses, powered by open table formats like Delta Lake, combine the best of both by bringing ACID transactions and BI performance to data lakes.
Delta Lake 4.3.0
Delta 4.3.0 deepens Unity Catalog integration by making it the source of truth for managed table operations via the UC Delta REST API, introduces replaceOn/replaceUsing DataFrame APIs for selective row-level data replacement, and improves UniForm with atomic Iceberg conversion and incremental metadata updates. Delta Sharing gains streaming support, Change Data Feed capabilities, and Trigger.AvailableNow, plus performance improvements like better V2 checkpoint parallelization and variant column statistics for data skipping.
EventsLTAP - Lake Transactional/Analytical Processing: a new data architecture that unifies OLAP and OLTP
LTAP (Lake Transactional Analytical Processing) is a new data architecture that unifies OLAP and OLTP storage, eliminating data copying and pipelines. It allows a single copy of data for both transactional and analytical systems, built on open formats like Postgres, Delta Lake, and Iceberg, without compromising performance.
What is data pipeline architecture?
Data pipeline architecture separates ingestion, transformation, storage, and serving into distinct layers, with ELT largely replacing ETL as the dominant approach. Databricks unifies batch and streaming pipelines on a single platform (Lakeflow + Delta Lake + Unity Catalog), eliminating duplicate infrastructure and governance gaps.
Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering
Octopus Energy achieved a 50x cost reduction in their margin data engineering pipelines by re-architecting on Databricks for UK MHHS regulation. They leveraged Delta Lake Change Data Feed and Databricks Serverless to process 48x more data at a fraction of the original cost, improving freshness from weekly to daily.
rust-v0.32.3 adds support for the variant data type in Delta Lake schemas. The release is otherwise documentation maintenance with no other user-facing changes.
PyArrow 21.0.0 is now required and brings preliminary variant type support; this release also fixes regressions in MERGE operations, partition column changes, S3 GIL contention, and Unity Catalog operations. New features include typed custom metadata support in CommitProperties and Musl arm64 wheels.
CommunityHow I Mastered System Design Interviews
This video teaches a six-step framework for mastering data engineering system design interviews, covering requirements gathering, pipeline design, data modeling, storage and file formats, data quality and observability, and pipeline resilience. It demonstrates how to apply this framework with practical examples and back-of-the-envelope calculations to justify design choices.
Fixes a MERGE operation memory regression and allows partition column changes during table schema overwrites. Adds nanosecond timestamp support and resolves Python import issues on Linux systems with 64KB pages.
Expanded interoperability with Unity Catalog Open APIs
Unity Catalog Open APIs now offer expanded interoperability, with external access to UC managed Delta tables in Beta and credential vending generally available with M2M OAuth support. External engines like Apache Spark, Flink, and DuckDB can now create, read, and write to UC managed Delta tables, leveraging Delta Lake's new catalog commits feature for safe concurrent writes and audibility.
The Rosetta stone of CPS: Claroty’s AI-powered library
Claroty's AI-powered CPS Library, built on Databricks Custom Agents and Delta Lake, automates entity resolution for 17M+ industrial and healthcare assets, solving the asset identity crisis where 88% of CPS devices lack exact product codes. This multi-agent AI system improves vulnerability attribution accuracy by over 25% and provides new security recommendations for over 56% of analyzed devices.
v0.32.0 provides performance improvements to log parsing, enhancements to the Datafusion TableProvider, and fixes for critical bugs affecting MERGE operations, partitioning, and vacuuming. The release upgrades Datafusion to version 53 and Arrow to version 58.
UnityCatalog 0.4.1
Unity Catalog 0.4.1 adds atomic write guarantees for REPLACE TABLE AS SELECT and Dynamic Partition Overwrite operations on UC Managed Delta Tables, plus a credential-scoped file system to prevent out-of-memory errors in long-running Spark sessions. The release introduces VARIANT datatype support and fixes a critical JWT validation bypass that could allow user impersonation, requiring authorization-enabled deployments to add issuer and audience configuration before upgrading.
Delta Lake 4.2.0
Delta 4.2.0 enables atomic REPLACE TABLE, RTAS, and DPO for catalog-managed tables, enhances streaming capabilities, and adds a Kernel-based Flink connector. The release makes Variant generally available, adds geospatial and collation support, and includes comprehensive security hardening.
NewsStop Guessing Table Health — Let These Dashboards Tell You
Databricks offers two dashboards for monitoring table health and access: the Table Access Advisor and the Table Health Advisor. These dashboards provide insights into table ownership, read/write patterns, staleness, optimization status, and underlying file structures, helping users identify ghost tables and ensure best practices.
TutorialsHow to Sync Lakebase Tables to Delta with Lakehouse Sync
Databricks demonstrates how to sync Lakebase PostgreSQL tables to Delta tables within a Databricks Lakehouse using the Lakehouse Sync feature. This process enables analytical workloads on data originating from Lakebase applications by leveraging Delta and Spark.
Delta Lake 4.1.0
Delta Lake 4.1.0 supports Apache Spark 4.1.0 and introduces conflict-free enablement of Deletion Vectors and Column Mapping on existing tables without blocking concurrent writes. The release requires Java 17 and Spark 4.0.1 or higher (dropping Spark 3.5), adds full catalog-managed table support in Delta Kernel for Unity Catalog integration, and fixes MERGE/INSERT struct expansion bugs.
Delta Lake 4.0.1
Delta Lake 4.0.1 introduces a breaking change renaming catalogOwned-preview to catalogManaged, adds Unity Catalog OAuth with automatic token refresh, and enables UC-managed Delta table creation. It also fixes a Spark 4.0.1 compatibility issue causing NoSuchMethodError in REORG TABLE APPLY (PURGE).
python-v1.3.1: read support deletion vectors, column mapping
This release adds read support for Delta Lake tables utilizing deletion vectors and column mapping. It also includes performance improvements for table scans and predicate pushdown, alongside better error messages for Unity Catalog and LakeFS.
This release introduces several API changes and integrates `delta_kernel` for improved stats parsing performance. It also fixes issues with schema evolution during merge operations and null handling in scalar extraction.
CommunityApache Spark Was Hard Until I Learned These 30 Concepts!
The video explains 30 key Apache Spark concepts, starting with a comparison to MapReduce to highlight Spark's in-memory processing and DAG-based execution model. It then details Spark's cluster architecture, job execution flow (driver, executors, tasks), and memory management within executor containers.
TutorialsDelta Lake Masterclass | Azure Databricks | PySpark | From Zero-To-Expert
This video provides a comprehensive masterclass on Delta Lake using Azure Databricks and PySpark, covering its core concepts, internal workings, and practical applications. It demonstrates how Delta Lake solves data lake problems like lack of ACID support, DML operations, and schema enforcement, and teaches features like time travel, concurrency control, and optimization techniques.
Events[Demo] Lakebase: Real-time Operational & Analytical Data on One Platform
Lakebase allows users to create synced tables in Unity Catalog, combining Delta Lake data with other sources for real-time operational and analytical use. These synced tables can be configured for one-off snapshots or continuous updates, enabling unified data access for applications and historical analysis.
UnityCatalog 0.3.0
UnityCatalog 0.3.0 adds support for Spark 4.0 and Delta 4.0 with new credentials and external locations APIs enabling flexible external storage management. The release includes Kubernetes deployment via Helm charts and fixes for Delta time-travel SQL queries.
NewsCrypto at Scale: Building a High-Performance Platform for Real-Time Blockchain Data
NewsScaling Identity Graph Ingestion to 1M Events/Sec with Spark Streaming & Delta Lake
NewsScaling Data Engineering Pipelines: Preparing Credit Card Transactions Data for Machine Learning
Delta Lake 4.0.0
Delta Lake 4.0.0 introduces Delta Connect for Spark Connect connectivity, catalog-managed tables in preview for catalog-controlled commits, full Variant data type support for semi-structured storage, and graduates Type Widening to enable column type changes without rewriting data. The release requires Apache Spark 4.0 and discontinues active development of Delta Standalone and related connectors (Flink, Hive) in 4.x, shifting them to maintenance-only mode.
Delta Lake 3.3.2
Delta Lake 3.3.2 improves table maintenance with a fix for stale checksum file cleanup and enhances Kernel compatibility with multiple JVM classloaders. The release also fixes BinaryType mapping in Flink for improved data type compatibility.
Delta Lake 3.3.1
Delta Lake 3.3.1 includes a fix allowing user-specified schemas on read when consistent with table schema and documentation updates for Row Tracking Backfill. Delta Kernel 3.3.1 fixes handling of non-uniform value types in map fields within delta commit files.
Delta Lake 3.3.0
Delta Lake 3.3.0 adds Identity Columns for automatic unique keys, VACUUM LITE for faster transaction log-based cleanup, and enables Row Tracking backfill on existing tables for row-level lineage tracking. UniForm Iceberg can now be enabled on existing Delta tables without data rewriting, and Type Widening is now supported in Delta Kernel for reading type-evolved tables.
Delta Lake 3.2.1
Delta Lake 3.2.1 adds RESTORE support for clustered tables and enables Uniform Iceberg on existing tables via ALTER TABLE, eliminating costly data rewrites. Key fixes include string comparison with Unicode surrogate pairs, decimal and timestamp encoding in Parquet, and S3DynamoDB VACUUM throttling issues.
EventsAnnouncing Delta Lake 4.0 with Liquid Clustering. Presented by Shant Hovsepian at Data + AI Summit
EventsAnnouncing DuckDB Support for Delta Lake and a DuckDB Extension to Unity Catalog - Hannes Mühleisen
EventsLakehouse Format Interoperability With UniForm. Shant Hovsepian presents at Data + AI Summit 2024
Delta Lake 4.0.0 Preview
Delta Lake 4.0.0rc1 introduces support for Spark Connect, Type Widening, the Variant data type, and Coordinated Commits for flexible multi-cloud writes. This preview also includes fixes for liquid clustering, improved CDF query filter pushdown, and performance enhancements.
Delta Lake 3.2.0
This release introduces Liquid clustering for incremental optimization and preview support for Type Widening to alter column types without data rewrites. It also adds preview support for Apache Hudi in Delta UniForm tables and improves VACUUM operations with inventory tables and writer protocol checks.
