Summary
Databricks Lakehouse Federation enables teams to query live Backstage operational data directly alongside Unity Catalog billing tables with zero ETL data movement. Learn how to configure the required Postgres authentication workaround for Lakebase and leverage isolated compute to monitor the exact DBU cost of ephemeral development branches.
Summary generated by brickster.ai. For the full article, follow the source link above.
Topics
More from Databricks Blog
Solving defense supply chain visibility through governed data sharing
Governed data sharing enables defense supply chain visibility across 200,000+ suppliers by allowing every tier to publish validated evidence directly from existing records without exposing source systems. Powered by open protocols and existing reporting obligations, this federated approach eliminates manual data calls while avoiding vendor lock-in and costly platform overhauls.
Announcing Workday Data Connect federation in Unity Catalog
The new Workday Data Connect connector (Beta) brings zero-copy federation to Unity Catalog, allowing teams to query Workday's shared Iceberg tables directly from cloud storage without ingestion pipelines or data duplication. Queries run on Databricks compute under Unity Catalog governance, enabling you to combine live HR and financial data with existing Databricks datasets for Genie-powered exploration, workforce analytics, and AI.
Introducing Funke: Native HL7v2 Parsing on Databricks
Funke is an open-source Python and PySpark library that parses HL7v2 electronic health record messages directly into native Spark types on the Databricks Lakehouse while preserving their complete message hierarchy. Rebuilt around Unity Catalog, Declarative Automation Bundles, and Spark Declarative Pipelines as the successor to Smolder, it includes a runnable demo to help you stand up an end-to-end HL7 ingestion pipeline in minutes.
Lakebase and Agentic SDLC: Branching Databases for Coding Agents
Lakebase resolves the database bottleneck for parallel coding agents by providing sub-second, scale-to-zero copy-on-write database branching for each agent. Learn how to implement an end-to-end workflow pairing Claude Code, Git worktrees, and GitHub Actions to run Drizzle migrations, deploy preview environments on Databricks Apps, and test against Unity Catalog-masked data.
Biomedical Imaging's Real Bottleneck Is the Data, Not the Model
The primary bottleneck in medical imaging AI is fragmented data infrastructure rather than model architecture. Scaling clinical and R&D impact requires a governed lakehouse foundation to centralize imaging assets, de-identify scans at scale, and link them directly with EHR, omics, and trial data.
Load terabytes of data in minutes into Lakebase Postgres
Lakebase Postgres leverages an LTAP architecture to offload bulk loading to Spark, building pages and indexes in parallel to load terabytes of data up to 147x faster. By writing directly to storage and publishing the final manifest through a compact WAL record, the system avoids consuming live application resources and keeps OLTP queries unaffected.