Unity Catalog
Recent items mentioning Unity Catalog across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
What is Unity Catalog?
Unity Catalog is the governance layer built into Databricks. It puts every data and AI asset in your account, including tables, views, volumes of files, functions, and ML models, into one three-level namespace (catalog.schema.object) and gives you a single place to control who can access what. Access rules are enforced across your workspaces automatically.
The problem it solves is scattered governance. Without a central catalog, each workspace keeps its own permissions, and there's no shared record of what data exists, where it came from, or who used it. Unity Catalog adds access control down to individual rows and columns, automatic lineage tracking, audit logging, tagging, data quality monitoring, and secure data sharing with other organizations, all in one system.
It's now the assumed foundation of the platform. Workspaces created after November 8, 2023 have it enabled automatically, while older workspaces need a manual upgrade from their legacy metastore. There's also an open source implementation on GitHub, and features such as metric views only work in workspaces enabled for Unity Catalog.
Do I need to enable Unity Catalog?
If your workspace was created after November 8, 2023, it's already enabled automatically. Older workspaces need a manual upgrade, and platform features such as metric views require a workspace enabled for Unity Catalog, so upgrading is usually the practical choice.
What does Unity Catalog actually govern?
Tables, views, volumes for non-tabular files, functions, ML models, and services, all organized in a catalog.schema.object namespace. It also manages securables like storage credentials, external locations, connections, and shares.
Is Unity Catalog open source?
There's an open source implementation of Unity Catalog available on GitHub. The version built into Databricks is managed for you and comes enabled on new workspaces.
Does Unity Catalog work across multiple workspaces?
Yes. It operates automatically across workspaces, enforcing access control, tracking lineage, and logging activity for auditing in one place rather than per workspace.
Sources: What is Unity Catalog? (Databricks docs) · Get started with Unity Catalog (Databricks docs) · Create a metric view, prerequisites (Databricks docs)
Multi-statement transactions in Unity Catalog now enable atomic writes across both Delta and Iceberg tables, eliminating custom reconciliation logic and data copying between systems 2. Meanwhile, catalog interoperability has expanded with Google Cloud's Borderless Lakehouse to allow zero-duplication data sharing with BigQuery 4, alongside new OpenSharing integrations that bring AWS HealthLake clinical data directly under Unity Catalog governance 8.
Generated daily from the 10 most recent items mentioning Unity Catalog. Click any [N] to jump to the source.
We wrote about this
- Genie One MCP server: setup and migrationHow to connect Claude Code, Claude, Cursor and ChatGPT to the GA Genie One MCP server, and what to change before the Beta endpoint reaches end of life on 30 November 2026.14 min read
- Databricks put agent skills in Unity Catalog, and gave them two namesA skill is now a catalog object you grant like a table. What the Beta does, the download-versus-live decision it forces, and why the docs cannot decide what to call it.6 min read
- What is Databricks Governance Hub, and what does it actually do?One account-level screen for data health, AI usage, cost, and tags across your whole Databricks estate: what each page shows, how to turn it on, and what is coming next.7 min read
- What is Genie Ontology, and how does it actually work?Half of it is defined on purpose. The other half assembles itself from dashboards and queries you already have. When definitions conflict, Genie weighs their authority by a ranking whose weights are not published, and may ask you to clarify.10 min read
TutorialsMigrate Databricks classic workloads to serverless with Genie Migrate | Databricks Demo
Genie Migrate is a Databricks tool designed to help users migrate workloads, including notebooks and jobs, from classic clusters to serverless. The tool automatically analyzes code and cluster infrastructure, identifies compatibility issues, and provides suggestions or applies fixes to facilitate the transition to serverless environments.
ReleasesMulti-Statement Transactions in Unity Catalog: Atomic Writes on Delta and Iceberg
Multi statement transactions in Unity Catalog allow atomic writes across multiple Delta or Iceberg tables, ensuring that updates either succeed completely or are aborted together. This capability eliminates the need for complex reconciliation logic, separate governance models, or data copying between systems.
TutorialsHow to Reduce AI Agent Costs | Unity Gateway Insights
Unity Gateway Insights helps organizations track AI agent usage and costs. The tool identifies common financial inefficiencies such as failing MCP tools, frontier models performing routine tasks, and repeated work.
Advancing Catalog Interoperability with Unity Catalog and Google Cloud’s Borderless Lakehouse
Customers can now access the same copy of data from either BigQuery or Databricks without duplication. This collaboration advances interoperability between Unity Catalog and Google Cloud's Borderless Lakehouse, establishing a long-term vision of shared catalog metadata.
The new Azure Databricks on Azure Government is now in Beta
Azure Databricks on Azure Government is now in Beta, bringing commercial platform parity with Unity Catalog governance, serverless Databricks SQL analytics, and U.S.-only data residency. Upcoming releases on the path to General Availability will add AI capabilities like Genie, Agent Bricks, and Model Serving, with FedRAMP Class D and IL5 authorization expected by early 2027.
What’s something you believed about Databricks for way too long before finding out you were wrong?
I’ll go first with two. For an embarrassingly long time I ran scheduled jobs on an all-purpose cluster because it was already up and I figured compute is compute. Nobody told me job clusters are billed at a lower rate. Found out when someone asked why our bill looked the way it did. Second one: I assumed Auto Loader schema evolution just quietly picked up new columns. It does, but only after the stream fails first and you restart it. Learned that from a 2am alert, not from the docs. Neither is complicated. I just never had a reason to question them until something broke. What’s yours? Could be Spark, Delta, Unity Catalog, billing, whatever. Bonus points if you found out in production. submitted by /u/UnluckySugar610 [link] [comments]
Clean data was never enough: How IFS Copperleaf taught AI its business with Databricks Genie One
IFS Copperleaf built a company-wide AI assistant on Databricks Genie One, using Genie Ontology and Unity Catalog Skills to establish shared business logic across nearly 30 disparate systems. By enabling users to query raw data across tools like Salesforce and M365 in plain English, the deployment achieved a 70% activation rate and covered its running costs in four months.
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
A new OpenSharing integration connects AWS HealthLake FHIR data directly to Databricks as a shared catalog, eliminating the need for costly export pipelines and data duplication. Healthcare organizations can now govern clinical data through Unity Catalog, run queries across notebooks and dashboards, and explore insights in natural language with Genie One.
Tabular foundation models on Databricks AI Runtime
Tabular foundation models accelerate predictive ML workflows by generating predictions on new datasets without requiring task-specific training. Databricks AI Runtime enables teams to benchmark and fine-tune these models against existing baselines using managed GPUs, MLflow tracking, and Unity Catalog integration.
NewsDistill a 120B LLM into a 0.8B Model on Databricks
LLM distillation teaches a smaller student model on answers from a larger teacher model using a narrow task at high volume, saving cost and improving speed while maintaining accuracy. This Databricks tutorial uses Foundation Model APIs, Serverless GPUs, MLflow evaluate, and Model Serving to build and deploy the smaller model end to end.
Lakeflow Declarative Pipelines with external: managed tables, or publish step?
I'm working on an Azure Databricks project where we must expose Delta tables through stable ADLS paths so external engines like Snowflake, Trino, and delta-rs apps can read directly from storage. We'd like to use Lakeflow, but managed streaming tables and materialized views live in Unity Catalog managed storage. Can we safely expose their physical paths like regular external Delta tables, or is path-based access unsupported? The alternative seems to be external Delta tables as sinks, but sinks look limited: Python/streaming only, no expectations or Auto CDC directly, and full refresh doesn't clear prior data. For this requirement, what architecture would you recommend: external sinks, a publish step, or something else? submitted by /u/warleyco96 [link] [comments]
Governed Spreadsheets on Databricks: Row Zero Acquisition & What It Means for Business Users on Databricks (Demo + Chat)
Hey everyone! Had the wonderful opportunity of recording a demo + chatting with Breck, the founder & former CEO of Row Zero, a governed spreadsheets platform Databricks just acquired. For business users, this will pair up nicely alongside the Genie One / Genie Agents experience. Some quick answers to common questions the video answers (used AI + transcript for them): When did Databricks acquire Row Zero? =The acquisition was announced in September 24, 2026. Can Databricks customers use Row Zero today? =Yes, with a Row Zero account. A built-in Databricks experience doesn't have a date yet. How will it be priced? =Not final. Expected to follow the rest of Databricks (pay for what you use), not per seat. Does it respect Unity Catalog permissions? =Yes. Row Zero queries Databricks as the signed-in user (OAuth recommended, service principals supported), so row-level security carries through. Can you write back to Databricks? =Yes, to tables with a primary key. You review a diff before changes apply. Can you export back to Excel? =Yes, to Excel or CSV. Exports over about 1 million rows get truncated because of Excel's row limit, and admins can turn export off entirely. All roads in data begin and end with a spreadsheet after all! PS: Always love to hear your feedback on the video. submitted by /u/JosueBogran [link] [comments]
Deploy models pattern via MLflow deployment jobs + multiple workspaces
Hi, I'm quite new to MLOps and need some hints for setting up what Databricks calls "Deploy model" pattern ( https://docs.databricks.com/aws/en/machine-learning/mlops/deployment-patterns ). Our Data Science team currently trains ML models in dev environment, in our dev Databricks account/workspace. They have direct access to production data, and the trainingㅤ is quite expensive. We'd like to train the model once and then promote it to production, which lives in a separate Databricks account/workspace. My idea to handle the deployment was to use MLflow deployment jobs ( https://docs.databricks.com/aws/en/mlflow/deployment-job ) and just allow the Data Science team to register the model in the production Unity Catalog. After the registration, the deployment job would kick in, do whatever else is needed, and then stop at an approval step. However, it turned out they can register the model version in the Unity Catalog from the dev workspace, but the deployment job doesn't run automatically. That makes the whole process suboptimal, as they now have to ping me to run the deployment job manually each time they register a new model version. I did some testing and the sole reason of that behavior seems to be the fact that the model comes from a different workspace, which doesn't "see" the attached deployment job that lives in the production workspace. Am I missing something here, or is it designed to work like that? What are the alternatives? submitted by /u/AvaPL [link] [comments]
Federated PostgreSQL timestamps in Unity Catalog: a view that gives everyone the same answer
How do you manage table schema changes in Databricks across dev/test/prod?
We run Databricks with Unity Catalog, and I'm trying to understand how other teams handle the lifecycle of managed tables (not pipelines or jobs, just the tables themselves). What I keep seeing: CREATE TABLE IF NOT EXISTS at the top of Job A DDL script in the repo that someone runs by hand Terraform for some tables and a different approach for the rest A few genuine questions: Where does the table definition live in your setup: in the pipeline code, in a separate repo, or only in the catalog? How do you review a schema change before it reaches prod? How do you roll the same change through dev, test, and prod? submitted by /u/ninilich [link] [comments]
Genie ignored my Unity Catalog wiki, so I tested what it actually reads
Last week I tried the new Unity Catalog wikis (Beta). I wrote a business rule into a table's wiki ("revenue = total_price of orders with status F only") and asked a Genie agent for revenue. It summed every order. When I asked who owns the table, it suggested checking a "wiki page outside this environment". I asked about it on the Databricks Community, and someone from Databricks replied: Genie agents don't use wikis as context today (no timeline yet). They read table and column comments. For a rule like this, use a metric view or a SQL expression, and keep text instructions for clarifications. So I tested that. Setup, in my own Azure workspace with the TPC-H sample data: - `samples.tpch.orders` copied into a table, all years - I asked about 1995 because it has F, O and P orders. The right answer (F only) is 32.69B. All orders is 171.63B - 5 Genie agents, each with ONE kind of context - 3 benchmark questions per agent ("What was our revenue in 1995?", "How much revenue did we make in 1995?", "What was our revenue in Q2 1995?"), run once in Agent mode and once in Chat mode Results: Context Agent mode Chat mode Nothing 0/3 0/3 Table + column comments 3/3 3/3 One text instruction 3/3 3/3 SQL expression (measure) + one example query 3/3 3/3 Metric view (filter inside the measure) 3/3 3/3 Things I noticed: - With no context, Agent mode wrote "This includes orders of every status." It states the assumption, but it doesn't ask. - The comments show up as the table and column descriptions under Configure > Sources. Two ALTER statements and it was fixed. - The measure editor wants table-qualified columns. `SUM(CASE WHEN status = 'F' THEN total_price END)` fails with "Table name or alias is required for column 'status'". `orders.status` works. - In Chat mode, Genie reused my example query word for word, `:year` parameter included. In Agent mode it didn't use the measure and wrote `WHERE status = 'F'` itself (still right). - The metric view agent used `MEASURE(revenue)` every time. The measure: ```yaml - name: revenue expr: SUM(total_price) FILTER (WHERE status = 'F') ``` Bonus: I also published a Unity Catalog Page called "Revenue". Genie One found it through the ontology ("Read page: Revenue"), answered 32.69B, said the definition "isn't certified yet" and listed the orders it left out. Best answer of the lot. Caveats: one run per mode, a simple rule, one table. Genie One also looked at an older table and agent in my workspace, but its reasoning says the Page decided the answer. And for a rule this simple even plain text worked, so this doesn't settle "structured context beats text" either way. Questions for you: - Has anyone seen text instructions break where a metric view or SQL expression held up? What kind of rule was it? - Where do you keep business rules for Genie today? Links: - The Databricks answer to my question: https://community.databricks.com/t5/data-governance/do-genie-agents-use-unity-catalog-wikis-as-context/td-p/175921 - Genie benchmarks: https://docs.databricks.com/aws/en/genie/benchmarks - Unity Catalog Pages: https://docs.databricks.com/aws/en/uc-semantics/pages submitted by /u/Zealousideal_Cry_889 [link] [comments]
How are you handling enterprise CDC and file ingestion in Databricks lately? (Lakeflow Connect vs. Custom Spark)
Hey everyone, I’ve been leaning heavily into Databricks Lakeflow Connect and Auto Loader (cloudFiles) recently for managing multi-source ingestion (pulling relational CDC streams like SQL Server alongside high-volume S3 file drops), and I wanted to see how the community is approaching this. Historically, setting up reliable ingestion meant writing custom PySpark streaming jobs, managing checkpoints manually, and dealing with schema evolution headaches when upstream source systems changed unexpectedly. Using Lakeflow Connect combined with Unity Catalog Volumes has noticeably streamlined how raw data hits the Bronze layer, especially for incremental file processing and maintaining clean lineage. I’m curious: Are you guys relying heavily on native Lakeflow/Auto Loader connectors for ingestion, or still building custom Spark/Delta streaming pipelines for specific edge cases? How are you handling schema drift when upstream source tables alter their structure? Any major pain points you've run into with CDC replication in Databricks so far? submitted by /u/SalamanderOpen859 [link] [comments]
How to Query Unity Catalog Views from External Engines
Genie agents and Unity Catalog wikis: what I tested, and what to use instead
Unity Catalog secrets: what each grant does, and the schema-level READ SECRET that fails
How Constellation is building a governed data foundation on Databricks
Constellation is building a governed Enterprise Data Platform on Databricks, using Unity Catalog to establish shared business definitions, lineage, and access control across domains. This standardized foundation directly supports the development of cross-functional AI use cases in nuclear engineering, outage planning, and sales.
NewsCompare Amazon RDS for Postgres with Databricks Lakebase
Lakebase provisions databases much faster, scales and idles automatically, branches instantly, and integrates with Unity Catalog seamlessly compared to Amazon RDS. Lakebase also enables zero-pipeline reverse ETL through continuous synced tables, whereas Amazon RDS requires separate standalone services and manual configuration.
Genie in Slack is now Generally Available: Governed AI cowork, right where work happens
Genie in Slack is now generally available, allowing users to direct message or @mention the assistant to receive governed answers grounded in business context without leaving Slack. Every query runs under the asker's Unity Catalog and source permissions, with approved MCP connectors enabling Genie to work with files, run scheduled analyses, and update external tickets or documents.
Seeking feedback on my Databricks data engineering portfolio project
Hi everyone, I’m preparing to apply for junior/entry-level Data Engineer roles focused on Databricks, and I’d appreciate honest feedback on a portfolio project I’ve been building. It is a governed lakehouse project for a synthetic table-tennis e-commerce dataset: - Deterministic input contract and synthetic fixture with intentional duplicates, malformed records and quality issues. - Bronze ingestion designed around Databricks Asset Bundles, Auto Loader, append-only raw tables, lineage metadata and retained checkpoints. - Silver transformation with explicit validation rules and a quarantine layer. - Gold publication with privacy controls: only aggregated, non-PII cells that meet a k-anonymity threshold are published. - Unity Catalog / Terraform / dedicated service principal / least-privilege model. - Manual serverless Databricks Job with materialization followed by a read-only verifier. - 287 local tests covering contracts, transformations, bundle configuration, verifier logic and Terraform privilege rules. The current DEMO-001 cloud milestone produced: - Silver: 15 orders, 10 customers, 7 products - Quarantine: 6 expected rejected rows - Gold: €640 revenue, 5 orders, 0 refunded lines - Pre-suppression controls: €812.20, 13 orders, 2 refunded lines - Status: COMPLETED_WITH_QUARANTINE I have deliberately kept the scope honest: this is a fixed deterministic snapshot, not CDC, streaming, SCD, generic incremental processing, SQL Warehouse, Power BI, or a production monitoring setup. Repository: tt-ecommerce I would especially value feedback on: Does this demonstrate relevant Databricks/Data Engineering capability? What is missing before using it in applications? Would you prioritise finishing governance/consumer access, adding SQL Warehouse/BI, or CI/CD and operational monitoring? Is the documentation clear, or does it feel over-engineered for a portfolio project? What would you expect to see in a strong junior Databricks project that is not here yet? Thanks — direct criticism is welcome. submitted by /u/Alone-Cow-1563 [link] [comments]
Databricks Metric Views are coming back to Power BI! [Private Preview]
First I'd like to give a bit of background because if you feel like you've heard about this feature before, you're right. Previous attempt to bring Metrics Views to PBI was "blocked" when MS removed the required opiton from the connector. In February 2026 - the Power BI Databricks connector introduced Metric View BI Compatibility Mode (Beta). It allowed Power BI to query Unity Catalog Metric Views, with Databricks rewriting the generated SQL to handle measures correctly. In April , Microsoft removed that option from the connector. Reports relying on it stopped working. Now, in November , Databricks has announced a different approach. In new approach Databricks pushes metric view to Power BI semantic model, so ach metric view becomes a table, dimensions become columns, and measures are translated into DAX. You can publish the model from Catalog Explorer or automate it using a Lakeflow Job. There are a few limitations: - Synchronization is one-way, so changes made in Power BI don't flow back to Unity Catalog. - Import mode also changes the security model - imported data is governed by Power BI permissions. For me it's a long awaited feature. I've never been a fan of defining the same business metrics separately in different BI tools. In my opinion metric definitions should live as close to the data as possible, in a shared layer that different reporting tools can use. Connect metric views to Power BI - Azure Databricks | Microsoft Learn https://preview.redd.it/zysb9za0o51i1.png?width=1355&format=png&auto=webp&s=ff3ce75b6bd2c25735f1c368a0721f09cbed8627 submitted by /u/szymon_dybczak [link] [comments]
Air gapped, not data dark: how Bright Machines gives customers live factory dashboards without opening the plant
Bright Machines replaced a fragile nine-service pipeline by using Databricks Unity Catalog federation with AWS Glue to query edge-captured Apache Iceberg data from air-gapped plants in place. This unified architecture makes factory PostgreSQL changes queryable within two minutes, powering 10-minute AI/BI Dashboard refreshes and zero-copy sharing under a single governed platform.
NewsDATA + AI LIVE: Context Is Just the Beginning
Databricks announced the next generation of Genie, an autonomous AI assistant that integrates structured and unstructured data using an automatically generated enterprise ontology. The video demonstrates how Genie utilizes Unity Catalog and over one hundred managed connectors to provide contextual insights and automate tasks across enterprise applications.
TutorialsCurate and govern your Genie Ontology
Genie ontology is a live map of your business that stays current as your organization changes, allowing Genie to answer questions with better accuracy, latency, and token efficiency. Genie search retrieves data in these sources and automatically creates and maintains one holistic search index across them instead of leaning on each provider's siloed search MCP.
Announcing GA of Genie Ontology
Genie Ontology is now Generally Available, delivering an agentic ontology that unifies business context across agents alongside the GA of Unity Catalog Domains and Pages. The release introduces Genie Search and Ontology Manager for curation, extending learned context to BI tools and directories while achieving a 91% pass rate on enterprise benchmark tasks using less than half the tokens.
From enterprise context to action: Genie Agents now analyze more data sources to transform your work
Genie Agents now support analyzing unstructured documents in Unity Catalog volumes with page-level citations in Public Preview, alongside generally available native MCP connectors to read and take action across tools like Jira, Slack, and Google Drive. Teams can also leverage new beta support for OpenSharing and Genie Clean Rooms to securely share governed agents with external partners and customers without exposing the underlying data.
From AWS Glue Catalog to Databricks Unity Catalog: An End-to-End Migration Guide
Catalog Linked Database - Snowflake writes UC Managed Iceberg tables to Unity Catalog
Is Unity Catalog REST API only accessible via public internet from Snowflake when attempting to set-up a private link connection (catalog integration) to a Unity Catalog (Catalog Linked Database)? I need a private link connection between Databricks and Snowflake. The use case is Snowflake needs to be able to write Managed Iceberg tables to Unity Catalog. submitted by /u/RazzmatazzLiving1323 [link] [comments]
I tried to break my own open-source safety check for agent writes on Databricks: 5 SQL attacks, a 2M-row Delta table, 2 real bugs found
A pipeline can run green and still ship wrong data. With AI agents writing more of our code, I wanted a check before changes reach production. https://preview.redd.it/3w9kidmk8w0i1.png?width=1280&format=png&auto=webp&s=99f764348291e6a0c2a31dc1cb6d2c7028717153 bricksentry (free, open source) tells you three things before a change ships: • Who depends on this table (Unity Catalog lineage) • What the change does to the data (rows added, removed, duplicated, totals moved) • How to undo it (the exact Delta restore point) Created by Author. Created by Author. Then it says OK, warn, needs a human, or block. I tried to break it on a 2M-row table. Production was never touched, and I fixed three bugs along the way. Created by Author. Website and docs: bricksentry.org Code (Apache-2.0): github.com/bricksentry/bricksentry Package: pypi.org/project/bricksentry If you’re running agents against Databricks today, I’d like to hear how you handle change-safety, and what breaks when you run bricksentry on your own tables. submitted by /u/According-Future5536 [link] [comments]
Unity Catalog Metric Views, now in Power BI: one unified, governed semantic layer for all your reporting and AI
Unity Catalog metric views can now be published directly to Power BI semantic models, establishing a single governed semantic layer across Power BI, AI/BI dashboards, and Genie. Teams can define data models and relationships once in Databricks, maintaining Unity Catalog access controls through DirectQuery while task orchestration keeps Power BI models in sync.
NewsDQX Studio Demo: No-Code Data Quality on Databricks
DQX Studio is a native Databricks application that enables business users to define, manage, and monitor data quality rules with AI assistance. It features a rules registry, table mapping, rule marketplaces, and drill-down reporting, and is available for installation through the Databricks Marketplace.
Accelerate GPU training on Unity Catalog tables with UCDataset
Now in beta, UCDataset is a native data loader that streams data directly from Unity Catalog tables to GPUs, accelerating training times by 10x and reducing compute costs. It plugs directly into standard PyTorch training loops to handle multi-GPU data sharding without custom code while ensuring automatic governance.
NewsServe Lakehouse Data to Your App with Lakebase Synced Tables
Lakebase synced tables allow you to effortlessly bring Delta tables from your lakehouse into your app's PostgreSQL database. The video demonstrates how to create a continuous sync pipeline and shows that data updates propagate from the lakehouse to the application within seconds.
Turn buying signals into pipeline with Sendoso gifts in Genie One
Sendoso's MCP server is now available on Databricks Marketplace, allowing teams to convert intent signals into previewed, approved gifts directly within Genie One conversations. Governed through Unity Catalog and Unity Gateway for access control and call logging, the integration keeps gift histories in the same workspace as pipeline data to track deal influence.
Databricks Asset Bundles now deploy registered model aliases to Unity Catalog and resolve bugs that repeatedly reapplied permissions and grants to recreated resources. The CLI also restores Databricks-specific pip magic execution in Jupyter notebooks over SSH and removes the CRITICAL compute priority class from AI Runtime.
Databricks Unity Catalog JDBC Connection: Can properties reference Databricks Secrets
This release introduces resources and data sources for LakeWatch, adds Unity Catalog secret referencing in databricks_connection, and supports Slack and Teams notification destinations. Bug fixes address provider client cache data races during concurrent operations and resolve post-apply drift in databricks_pipeline resources.
Job completion triggers just hit Beta, and they replace a pattern I've built too many times!!
The usual way to chain jobs across teams has always been a bit ugly. Either a parent job with Run Job tasks that has to own everything downstream, or a schedule set 30 minutes after the upstream job usually finishes, which works until the day it doesn't. Databricks just added a job completion trigger in Beta. A job can now start when one or more upstream jobs finish with a matching result, and the release notes say those upstream jobs can live in other workspaces on the same Unity Catalog metastore. A few things from the docs worth knowing before you try it: - You can add a SQL condition, so the downstream job only runs when the data is actually ready, not just when the upstream job exited. - Unlike the other trigger types, it doesn't drop completions when the job is at max concurrency. It holds them until a run can start. - It's part of the multiple triggers Beta, so a job can have up to five triggers, and a workspace admin has to turn it on in Previews. Where I'd use it: a gold job that waits on silver jobs owned by another team, without either team having to own the other's orchestration. That handoff has been a friction point on every multi-team platform I've worked on. It's Beta, so I wouldn't move critical prod chains onto it yet. I'd start testing it in dev now though. submitted by /u/OkImprovement7010 [link] [comments]
Do Genie agents use Unity Catalog wikis as context?
Databricks Apps: Are you using them in production yet or keeping it to internal prototypes?
Hey everyone, I’ve been diving into Databricks Apps recently and wanted to see how others are adopting it in their organizations. Historically, whenever we wanted to spin up a Streamlit app or a custom Python dashboard connected to our data in Databricks, it meant dealing with a headache of external infrastructure—setting up App Service or EC2, dealing with network security groups, configuring service principals, and managing separate auth flows. Being able to run Streamlit/Dash apps directly inside the Databricks workspace, backed by Unity Catalog for governance and serverless compute, feels like a massive quality-of-life improvement for internal tooling. I’m curious to hear from anyone who has put them into practice: Are you using Databricks Apps for production-grade tools, or mostly for quick internal MVPs and demos? How are you managing your CI/CD deployments for these apps? Have you run into any weird gotchas with state management, scaling, or dependencies? submitted by /u/SalamanderOpen859 [link] [comments]
How much "ZeroOps" are you actually seeing in Databricks lately? (Serverless, DLT, Unity Catalog)
Hey everyone, I’ve been looking into the push toward "ZeroOps" within the Databricks ecosystem—specifically leaning heavily into serverless compute, Unity Catalog automation, and declarative tools like DLT/Lakeflow to reduce infrastructure maintenance. Marketing terms aside, I'm curious how much of this is translating to reality in production environments. For those of you managing large Databricks workspaces: Have you fully transitioned to serverless compute (SQL and notebooks), or are you still tied to classic clusters for specific workloads? How much time has Unity Catalog actually saved your team regarding access control and data governance overhead? Where does the "ZeroOps" promise break down for you in day-to-day operations? submitted by /u/SalamanderOpen859 [link] [comments]
Has anyone built production pipelines with Databricks Lakeflow Designer yet? Initial thoughts?
Hey everyone, I’ve been exploring Databricks Lakeflow Designer recently and wanted to see how the community feels about it. For context, it’s Databricks’ new visual, low-code authoring tool built on top of Delta Live Tables (DLT) for building ingestion and transformation pipelines. Historically, I’ve always written DLT pipelines purely in Python or SQL via notebooks or Git folders. When I first saw a visual designer, I was a bit skeptical (usually worried about "black box" code or messy JSON metadata under the hood). However, seeing that it actually compiles down to standard DLT/Spark and integrates natively with Unity Catalog lineage is pretty slick. I’m curious: Are you guys using Lakeflow Designer for production workloads, or keeping it strictly for quick ingestion and prototyping? How does it handle complex CI/CD workflows compared to traditional repo-based DLT code? Any gotchas or limitations you've run into with it so far? submitted by /u/SalamanderOpen859 [link] [comments]
Lakebase streaming sync is now GA
Hey Reddit, I’m happy to share one more new feature. The Lakebase streaming sync is now officially GA. Lots of customers are already using this feature to write streaming data with latency in 10s of milliseconds to Lakebase. It's much simpler to use than foreachBatch and you don't have to worry about connection pools, rate limits, error handling, etc. Here's what's new since we announced the Public Preview earlier this year: Declarative merge strategies*: Choose, per column, how an incoming row is reconciled with the existing row on a primary-key conflict. For example, you can keep the latest non-null value, keep the first value written, or take the value from the row with the latest event time. No custom SQL is required. External PostgreSQL via UC connections (Public Preview)*: Write to Postgres databases outside Lakebase, authenticated through a Unity Catalog connection so no credentials live in code. INTERVAL types*: DayTimeInterval and YearMonthInterval columns map to Postgres interval on DBR 19 and above. Serverless support: The sink now runs on serverless compute for notebooks and jobs, using the AvailableNow trigger. \ Merge strategies, external Postgres, and interval types require DBR 19.* Example: df.writeStream .format("postgresql") .outputMode("update") // optional; can be append mode as well .option("upsertkey", "primaryKey") // optional .option("upsert.columnStrategies", "last_updated_at:greatest,status:latestBy(last_updated_at)") .toTable("catalog.schema.table") SDP support remains in Private Preview. Public Preview, which adds support for native Lakebase streaming tables, is slated for later this year. submitted by /u/inkypinky [link] [comments]
How I check every prompt before the LLM on Databricks: GovTech's LionGuard on a small CPU endpoint
I wanted one shared safety check that every agent, job or SQL query on Databricks calls before the LLM. How it works: - LionGuard 2 Lite from GovTech Singapore: a small classifier on Google's EmbeddingGemma. It scores insults, hate, violence, self-harm and misconduct, and it handles English, Singlish, Chinese, Malay and Tamil. - Packaged as one MLflow model, registered in Unity Catalog, served on a small CPU endpoint. - From SQL, ai_query sends each question to the guard. Only questions with an unsafe score under 0.5 go to the LLM. Results on 7 synthetic questions: a Singlish insult, a Malay threat, a harmful Chinese message and an English "how do I hack the website" question were blocked. Safe questions in English, Singlish and Tamil were answered. Lessons: - The full set-up took 25 minutes. 19 of them were the endpoint start. - Do not use scale to zero for a production guard (cold start). - Turn on inference tables from day one, so you can show what was blocked. - It does not detect prompt injection. That needs a separate check. How do you check prompts for your agents today? Built-in AI Gateway guardrails, a custom endpoint, or both? submitted by /u/sundarleads [link] [comments]
[Beta] OneDrive connector in Lakeflow Connect now available!
Hey r/databricks , The Lakeflow Connect connector for Microsoft OneDrive (for business) is now available in Beta . You can ingest structured and unstructured files from a user's "My files" drive into Delta tables for analytics and AI workloads. Managed pipeline authoring is available through the Lakeflow Pipelines API or Declarative Automation Bundles (DABs) . UI-based pipeline authoring is not supported at this time, but will be available soon! Managed pipelines support incremental ingestion, automated retries, Unity Catalog governance, schema evolution for supported formats, and OneDrive-specific and standard file metadata. For direct control over ingestion logic, you can use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files , or COPY INTO pointed at OneDrive URLs. Common workloads Loading Excel, CSV, JSON, XML, Parquet, Avro, and ORC files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document for extraction, search, and agent workloads. Capturing file metadata without loading file contents. Examples using the Spark + SQL APIs Read an Excel range from OneDrive with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_onedrive_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load("https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents/Report.xlsx")) Incrementally ingest PDFs and Word documents, then parse them with ai_parse_document: CREATE OR REFRESH STREAMING TABLE onedrive_documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM onedrive_documents; CREATE OR REFRESH STREAMING TABLE onedrive_documents AS SELECT * FROM STREAM read_files( "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents", `databricks.connection` => "my_onedrive_conn", format => "binaryFile", pathGlobFilter => "*.{pdf,docx}"); Try the managed connector Ask a workspace admin to open Previews and enable Lakeflow Connect for OneDrive For Business . Create a Unity Catalog connection to OneDrive . Create a managed pipeline using Declarative Automation Bundles or the Lakeflow Pipelines API . The setup guide includes bundle YAML and an API example notebook . Schedule ingestion with Lakeflow Jobs. For DABs, define the scheduled job in the bundle. Current Beta scope The connector reads from a user's OneDrive for Business "My files" drive. For SharePoint sites, team sites, and shared document libraries, use the SharePoint connector . The connection authenticates as the signed-in Microsoft account and can ingest files that account can access. Managed ingestion requires Unity Catalog, Serverless compute, Databricks Runtime 19 or above, and the PREVIEW pipeline channel. DABs require Databricks CLI version 1.23.0 or above . Resources OneDrive managed connector documentation Managed pipeline setup: DABs and API examples Spark + SQL APIs and examples OneDrive connector limitations Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow If you try the Beta, please provide us your feedback! submitted by /u/BricksterJ [link] [comments]
Unity Catalog secrets: GRANT READ SECRET ON SCHEMA fails with PRIVILEGE_NOT_APPLICABLE_TO_ENTITY
How public sector organizations can govern AI at scale
Unity Gateway lets public sector teams govern AI the way they already govern data in Unity Catalog, with controls for model access, spend caps, and request-level tracking. Databricks also manages data residency for its hosted models by geography, giving compliance teams a documented answer on where AI requests are processed.
NewsDatabricks MemCurrent, Genie Agents, Azure MPC Server & Feature Views | AI Newsround - October 2026
Databricks announced the beta release of MemCurrent for persistent agent memory, along with Genie Flash mode for faster agent responses and general availability of Feature Views. Azure MCP Server 3.0 reached general availability, and several new AI models including Anthropic Claude Haiku 5.5 and Microsoft Decision 1 were introduced.
A builder’s guide to scaling Databricks Apps
Scaling Databricks Apps starts with agreeing on target response times and peak user counts, then routing heavy work to the right service while app compute handles the UI and coordinates calls. Keep code responsive and state recoverable by caching per user and storing hard state in Lakebase or Unity Catalog tables, then size instances, workers and Lakebase connections from load tests and telemetry.
NewsWhy the Creator of Delta Lake Says: Stop Writing ETL
Spark declarative pipelines, or SDP, allow engineers to define what data tables and transformations should exist rather than manually scripting how to execute them, letting the system handle the underlying orchestration, incremental updates, and error handling. Michael Armbrust and Alexandra Chashchina explain how SDP simplifies data pipelines across various workloads, improves cost efficiency, and reduces manual intervention compared to traditional Spark and Delta Live Tables approaches.
From Delta Sharing to OpenSharing: what changed, what's next
OpenSharing evolves Delta Sharing into an open, Linux Foundation-governed protocol that extends sharing beyond tables to AI assets such as agent skills and models, so teams can share across platforms, Iceberg engines, and on-premises storage without copying data or building custom pipelines per partner. It's additive: existing Delta Sharing setups keep working, and you enable the new asset types on a Unity Catalog metastore using the same share, recipient, and grant workflow.
Push-based CDC into Unity Catalog with Zerobus Ingest and Debezium
Debezium Server and Databricks Zerobus Ingest now stream CDC events straight into Unity Catalog Delta tables using a push-based architecture, with no intermediate Kafka brokers or pull-based connectors required. Because data is pushed outward, locked-down private networks that block inbound connections can still replicate to the lakehouse, which simplifies the infrastructure needed to build current-state tables for downstream analytics.
NewsDataBuddy By Tencent Cloud
Tencent Cloud DataBuddy is an agentic platform designed to automate and manage data engineering, governance, and analysis workflows within e-commerce operations. The platform enables users to execute complex tasks from data ingestion to trend analysis while maintaining human oversight at critical approval checkpoints.
Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation
Are people using Collibra to set owners of Tables in Unity Catalog?
Wondering if there's any value add of Collibra to Unity Catalog. Another tool to manage, wondering Databricks Request For Access is sufficient. How does your org manage access to datasets for technical and business users? Collibra or Databricks request for access or DABS? submitted by /u/RazzmatazzLiving1323 [link] [comments]
Backing up Unity Catalog volumes without melting the driver
Get Tuesday's version of this
Tracking Unity Catalog? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.