Delta Sharing

Delta Lake 3.3.3

Delta 3.3.3 fixes transaction log retention bugs that broke time travel and CDF reads, and a Delta Sharing deletion vector cache bug causing long-running queries to fail. It adds opt-in RANDOMIZE_FILE_PREFIXES to spread S3 object keys for high-throughput workloads and upgrades Delta Sharing client for improved OAuth and retry handling.

Workspace ID validation for unified-provider resources shifted from plan time to apply time, reducing unnecessary API calls and eliminating false positives for restricted credentials and dynamic workspace IDs. The release adds a databricks_recipients data source for Delta Sharing, trace_location support for MLflow experiment traces in Unity Catalog, and fixes for VIEW column comment updates and access control rule set drift detection.

Delta Lake 4.3.1

Delta Lake 4.3.1 fixes OAuth authentication failures in the Delta REST Catalog caused by incorrect key lowercasing and enables S3A fast listing when using OSS UnityCatalog's CredScopedFileSystem wrapper. It also prevents the reserved is_managed_location property from persisting into managed table metadata.

Delta Lake 4.3.0

Delta 4.3.0 deepens Unity Catalog integration by making it the source of truth for managed table operations via the UC Delta REST API, introduces replaceOn/replaceUsing DataFrame APIs for selective row-level data replacement, and improves UniForm with atomic Iceberg conversion and incremental metadata updates. Delta Sharing gains streaming support, Change Data Feed capabilities, and Trigger.AvailableNow, plus performance improvements like better V2 checkpoint parallelization and variant column statistics for data skipping.

Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era

When Databricks pioneered Delta Sharing in 2021, we set out to solve a problem that...

Sciene AI Companion: building an autonomous Customer Success platform on Databricks

Sciene built an autonomous Customer Success platform on Databricks, enabling AI-powered CSMs to standardize and scale work with context-aware emails, meeting decks, and account diagnostics. This end-to-end Databricks solution, leveraging Delta Sharing, Lakebase, and SQL Warehouses, significantly improved productivity and saved up to 6x time on key workflows.

Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3

Spatial SQL is now Generally Available on Databricks, bringing native geospatial data types, 90+ ST_* functions, and AI/BI Dashboards that render maps natively. This release also includes major performance improvements, open lakehouse support via Delta Sharing and Iceberg v3, and Apache Spark 4.2 compatibility for geo columns.

Unlocking SAP Business Context in Databricks with Semantic Metadata Delta Sharing

SAP Business Data Cloud now automatically syncs semantic metadata, including descriptions and key relationships, into Unity Catalog, making SAP data instantly AI-ready and more discoverable. SAP PersonalData governance tags are also automatically available in Unity Catalog, enabling fine-grained access controls with ABAC.

Stripe data now available on Databricks via Databricks Marketplace

Stripe data is now available on Databricks Marketplace, enabling you to activate a Stripe data pipeline with Delta Sharing in minutes and instantly power AI applications. Share Stripe payment and business data directly into Unity Catalog to create a single source of truth and query live payment data for models, agents, and Genie workspaces.

Delta Lake 4.2.0

Delta 4.2.0 enables atomic REPLACE TABLE, RTAS, and DPO for catalog-managed tables, enhances streaming capabilities, and adds a Kernel-based Flink connector. The release makes Variant generally available, adds geospatial and collation support, and includes comprehensive security hardening.

Delta Lake 4.1.0

Delta Lake 4.1.0 supports Apache Spark 4.1.0 and introduces conflict-free enablement of Deletion Vectors and Column Mapping on existing tables without blocking concurrent writes. The release requires Java 17 and Spark 4.0.1 or higher (dropping Spark 3.5), adds full catalog-managed table support in Delta Kernel for Unity Catalog integration, and fixes MERGE/INSERT struct expansion bugs.

SQL warehouses now support "5X-Large" cluster sizes and a higher maximum of 40 clusters. This release also fixes permanent drift for external model credentials in databricks_model_serving and improves dashboard file content change detection.

Delta Lake 4.0.0

Delta Lake 4.0.0 introduces Delta Connect for Spark Connect connectivity, catalog-managed tables in preview for catalog-controlled commits, full Variant data type support for semi-structured storage, and graduates Type Widening to enable column type changes without rewriting data. The release requires Apache Spark 4.0 and discontinues active development of Delta Standalone and related connectors (Flink, Hive) in 4.x, shifting them to maintenance-only mode.

Delta Lake 3.3.1

Delta Lake 3.3.1 includes a fix allowing user-specified schemas on read when consistent with table schema and documentation updates for Row Tracking Backfill. Delta Kernel 3.3.1 fixes handling of non-uniform value types in map fields within delta commit files.

Delta Lake 3.3.0

Delta Lake 3.3.0 adds Identity Columns for automatic unique keys, VACUUM LITE for faster transaction log-based cleanup, and enables Row Tracking backfill on existing tables for row-level lineage tracking. UniForm Iceberg can now be enabled on existing Delta tables without data rewriting, and Type Widening is now supported in Delta Kernel for reading type-evolved tables.

Delta Lake 3.2.1

Delta Lake 3.2.1 adds RESTORE support for clustered tables and enables Uniform Iceberg on existing tables via ALTER TABLE, eliminating costly data rewrites. Key fixes include string comparison with Unicode surrogate pairs, decimal and timestamp encoding in Parquet, and S3DynamoDB VACUUM throttling issues.

Delta Lake 3.2.0

This release introduces Liquid clustering for incremental optimization and preview support for Type Widening to alter column types without data rewrites. It also adds preview support for Apache Hudi in Delta UniForm tables and improves VACUUM operations with inventory tables and writer protocol checks.