Latest releases from the Databricks ecosystem.

This release adds support for the VARIANT data type.

This release fixes several regressions, including issues with MERGE operations, overwriting tables with partition changes, and schema mode with mixed-case columns. It also enables passing non-string datatypes in custom commit metadata and updates the minimum PyArrow version to 21.0.0 for preliminary variant type support.

This release fixes a bug preventing partition column changes when overwriting tables and addresses a memory regression in Python MERGE operations. It also adds support for passing non-string datatypes in custom commit metadata and enables nanosecond timestamp support.

This release improves the new Datafusion TableProvider and log parsing performance, alongside numerous bug fixes addressing issues like incorrect row counts for deletes, merge failures, and S3/Azure storage inconsistencies. A new feature allows adding post-commit hook properties to DeltaTable.restore operations.

python-v1.5.0: faster writes, log compaction, spil config in MERGE

This release introduces faster Delta table writes through parallel partition writers and adds log compaction for improved performance. The `get_add_actions` method now returns an Arrow Table instead of an Arrow RecordBatch, which is a breaking change.

This release provides a fix for a Rust-related issue, specifically backporting a previous pull request. Databricks users will see improved stability due to this underlying fix.

This release introduces a session-first DataFusion integration and exposes Delta Lake Vacuum (DV) metadata as Arrow streams. It also fixes issues with schema merge appends for generated columns and improves parquet predicate pushdown.

This release fixes an issue where file statistics were not aligned with the Parquet read schema and resolves DML predicates against the execution scan schema. It also updates the asserted nullability in the DataValidation output schema.

This release enables deletion vector features for Delta tables and improves logical planning for update and delete operations. It also fixes issues with reporting failed data in data checks and supporting user names in Azure URLs.

This release introduces a new table provider in the query builder and DataSink, along with migrating table scans. It also fixes an issue where deleting from an empty table failed and addresses data validation mask exhaustion.

python-v1.3.1: read support deletion vectors, column mapping

This release adds read support for Delta Lake tables utilizing deletion vectors and column mapping. It also includes performance improvements for table scans and predicate pushdown, alongside better error messages for Unity Catalog and LakeFS.

This release adds support for multiple constraints at once, generates Symlink Manifests for external engines, and introduces GCS auto-registration. It also includes fixes for schema evolution in merge operations, improved error reporting, and enhanced handling of empty tables.

This release introduces several API changes and integrates `delta_kernel` for improved stats parsing performance. It also fixes issues with schema evolution during merge operations and null handling in scalar extraction.