Skip to content
All topics

Unity Catalog

Recent items mentioning Unity Catalog across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

60 recent items2 releases17 news12 videos29 community threads

What is Unity Catalog?

Unity Catalog is the governance layer built into Databricks. It puts every data and AI asset in your account, including tables, views, volumes of files, functions, and ML models, into one three-level namespace (catalog.schema.object) and gives you a single place to control who can access what. Access rules are enforced across your workspaces automatically.

The problem it solves is scattered governance. Without a central catalog, each workspace keeps its own permissions, and there's no shared record of what data exists, where it came from, or who used it. Unity Catalog adds access control down to individual rows and columns, automatic lineage tracking, audit logging, tagging, data quality monitoring, and secure data sharing with other organizations, all in one system.

It's now the assumed foundation of the platform. Workspaces created after November 8, 2023 have it enabled automatically, while older workspaces need a manual upgrade from their legacy metastore. There's also an open source implementation on GitHub, and features such as metric views only work in workspaces enabled for Unity Catalog.

Do I need to enable Unity Catalog?

If your workspace was created after November 8, 2023, it's already enabled automatically. Older workspaces need a manual upgrade, and platform features such as metric views require a workspace enabled for Unity Catalog, so upgrading is usually the practical choice.

What does Unity Catalog actually govern?

Tables, views, volumes for non-tabular files, functions, ML models, and services, all organized in a catalog.schema.object namespace. It also manages securables like storage credentials, external locations, connections, and shares.

Is Unity Catalog open source?

There's an open source implementation of Unity Catalog available on GitHub. The version built into Databricks is managed for you and comes enabled on new workspaces.

Does Unity Catalog work across multiple workspaces?

Yes. It operates automatically across workspaces, enforcing access control, tracking lineage, and logging activity for auditing in one place rather than per workspace.

Sources: What is Unity Catalog? (Databricks docs) · Get started with Unity Catalog (Databricks docs) · Create a metric view, prerequisites (Databricks docs)

What's happening in Unity CatalogAI synthesis · updated Nov 19, 2026

Multi-statement transactions in Unity Catalog now enable atomic writes across both Delta and Iceberg tables, eliminating custom reconciliation logic and data copying between systems 2. Meanwhile, catalog interoperability has expanded with Google Cloud's Borderless Lakehouse to allow zero-duplication data sharing with BigQuery 4, alongside new OpenSharing integrations that bring AWS HealthLake clinical data directly under Unity Catalog governance 8.

Generated daily from the 10 most recent items mentioning Unity Catalog. Click any [N] to jump to the source.

Reddit

What’s something you believed about Databricks for way too long before finding out you were wrong?

I’ll go first with two. For an embarrassingly long time I ran scheduled jobs on an all-purpose cluster because it was already up and I figured compute is compute. Nobody told me job clusters are billed at a lower rate. Found out when someone asked why our bill looked the way it did. Second one: I assumed Auto Loader schema evolution just quietly picked up new columns. It does, but only after the stream fails first and you restart it. Learned that from a 2am alert, not from the docs. Neither is complicated. I just never had a reason to question them until something broke. What’s yours? Could be Spark, Delta, Unity Catalog, billing, whatever. Bonus points if you found out in production. submitted by /u/UnluckySugar610 [link] [comments]

00UnluckySugar6102d ago
Reddit

Lakeflow Declarative Pipelines with external: managed tables, or publish step?

I'm working on an Azure Databricks project where we must expose Delta tables through stable ADLS paths so external engines like Snowflake, Trino, and delta-rs apps can read directly from storage. We'd like to use Lakeflow, but managed streaming tables and materialized views live in Unity Catalog managed storage. Can we safely expose their physical paths like regular external Delta tables, or is path-based access unsupported? The alternative seems to be external Delta tables as sinks, but sinks look limited: Python/streaming only, no expectations or Auto CDC directly, and full refresh doesn't clear prior data. For this requirement, what architecture would you recommend: external sinks, a publish step, or something else? submitted by /u/warleyco96 [link] [comments]

00warleyco963d ago
Reddit

Governed Spreadsheets on Databricks: Row Zero Acquisition & What It Means for Business Users on Databricks (Demo + Chat)

Hey everyone! Had the wonderful opportunity of recording a demo + chatting with Breck, the founder & former CEO of Row Zero, a governed spreadsheets platform Databricks just acquired. For business users, this will pair up nicely alongside the Genie One / Genie Agents experience. Some quick answers to common questions the video answers (used AI + transcript for them): When did Databricks acquire Row Zero? =The acquisition was announced in September 24, 2026. Can Databricks customers use Row Zero today? =Yes, with a Row Zero account. A built-in Databricks experience doesn't have a date yet. How will it be priced? =Not final. Expected to follow the rest of Databricks (pay for what you use), not per seat. Does it respect Unity Catalog permissions? =Yes. Row Zero queries Databricks as the signed-in user (OAuth recommended, service principals supported), so row-level security carries through. Can you write back to Databricks? =Yes, to tables with a primary key. You review a diff before changes apply. Can you export back to Excel? =Yes, to Excel or CSV. Exports over about 1 million rows get truncated because of Excel's row limit, and admins can turn export off entirely. All roads in data begin and end with a spreadsheet after all! PS: Always love to hear your feedback on the video. submitted by /u/JosueBogran [link] [comments]

00JosueBogran3d ago
Reddit

Deploy models pattern via MLflow deployment jobs + multiple workspaces

Hi, I'm quite new to MLOps and need some hints for setting up what Databricks calls "Deploy model" pattern ( https://docs.databricks.com/aws/en/machine-learning/mlops/deployment-patterns ). Our Data Science team currently trains ML models in dev environment, in our dev Databricks account/workspace. They have direct access to production data, and the trainingㅤ is quite expensive. We'd like to train the model once and then promote it to production, which lives in a separate Databricks account/workspace. My idea to handle the deployment was to use MLflow deployment jobs ( https://docs.databricks.com/aws/en/mlflow/deployment-job ) and just allow the Data Science team to register the model in the production Unity Catalog. After the registration, the deployment job would kick in, do whatever else is needed, and then stop at an approval step. However, it turned out they can register the model version in the Unity Catalog from the dev workspace, but the deployment job doesn't run automatically. That makes the whole process suboptimal, as they now have to ping me to run the deployment job manually each time they register a new model version. I did some testing and the sole reason of that behavior seems to be the fact that the model comes from a different workspace, which doesn't "see" the attached deployment job that lives in the production workspace. Am I missing something here, or is it designed to work like that? What are the alternatives? submitted by /u/AvaPL [link] [comments]

00AvaPL3d ago
Databricks CommunityCommunity Articles

Federated PostgreSQL timestamps in Unity Catalog: a view that gives everyone the same answer

003d ago
Reddit

How do you manage table schema changes in Databricks across dev/test/prod?

We run Databricks with Unity Catalog, and I'm trying to understand how other teams handle the lifecycle of managed tables (not pipelines or jobs, just the tables themselves). What I keep seeing: CREATE TABLE IF NOT EXISTS at the top of Job A DDL script in the repo that someone runs by hand Terraform for some tables and a different approach for the rest A few genuine questions: Where does the table definition live in your setup: in the pipeline code, in a separate repo, or only in the catalog? How do you review a schema change before it reaches prod? How do you roll the same change through dev, test, and prod? submitted by /u/ninilich [link] [comments]

00ninilich4d ago
Reddit

Genie ignored my Unity Catalog wiki, so I tested what it actually reads

Last week I tried the new Unity Catalog wikis (Beta). I wrote a business rule into a table's wiki ("revenue = total_price of orders with status F only") and asked a Genie agent for revenue. It summed every order. When I asked who owns the table, it suggested checking a "wiki page outside this environment". I asked about it on the Databricks Community, and someone from Databricks replied: Genie agents don't use wikis as context today (no timeline yet). They read table and column comments. For a rule like this, use a metric view or a SQL expression, and keep text instructions for clarifications. So I tested that. Setup, in my own Azure workspace with the TPC-H sample data: - `samples.tpch.orders` copied into a table, all years - I asked about 1995 because it has F, O and P orders. The right answer (F only) is 32.69B. All orders is 171.63B - 5 Genie agents, each with ONE kind of context - 3 benchmark questions per agent ("What was our revenue in 1995?", "How much revenue did we make in 1995?", "What was our revenue in Q2 1995?"), run once in Agent mode and once in Chat mode Results: Context Agent mode Chat mode Nothing 0/3 0/3 Table + column comments 3/3 3/3 One text instruction 3/3 3/3 SQL expression (measure) + one example query 3/3 3/3 Metric view (filter inside the measure) 3/3 3/3 Things I noticed: - With no context, Agent mode wrote "This includes orders of every status." It states the assumption, but it doesn't ask. - The comments show up as the table and column descriptions under Configure > Sources. Two ALTER statements and it was fixed. - The measure editor wants table-qualified columns. `SUM(CASE WHEN status = 'F' THEN total_price END)` fails with "Table name or alias is required for column 'status'". `orders.status` works. - In Chat mode, Genie reused my example query word for word, `:year` parameter included. In Agent mode it didn't use the measure and wrote `WHERE status = 'F'` itself (still right). - The metric view agent used `MEASURE(revenue)` every time. The measure: ```yaml - name: revenue expr: SUM(total_price) FILTER (WHERE status = 'F') ``` Bonus: I also published a Unity Catalog Page called "Revenue". Genie One found it through the ontology ("Read page: Revenue"), answered 32.69B, said the definition "isn't certified yet" and listed the orders it left out. Best answer of the lot. Caveats: one run per mode, a simple rule, one table. Genie One also looked at an older table and agent in my workspace, but its reasoning says the Page decided the answer. And for a rule this simple even plain text worked, so this doesn't settle "structured context beats text" either way. Questions for you: - Has anyone seen text instructions break where a metric view or SQL expression held up? What kind of rule was it? - Where do you keep business rules for Genie today? Links: - The Databricks answer to my question: https://community.databricks.com/t5/data-governance/do-genie-agents-use-unity-catalog-wikis-as-context/td-p/175921 - Genie benchmarks: https://docs.databricks.com/aws/en/genie/benchmarks - Unity Catalog Pages: https://docs.databricks.com/aws/en/uc-semantics/pages submitted by /u/Zealousideal_Cry_889 [link] [comments]

00Zealousideal_Cry_8894d ago
Reddit

How are you handling enterprise CDC and file ingestion in Databricks lately? (Lakeflow Connect vs. Custom Spark)

Hey everyone, I’ve been leaning heavily into Databricks Lakeflow Connect and Auto Loader (cloudFiles) recently for managing multi-source ingestion (pulling relational CDC streams like SQL Server alongside high-volume S3 file drops), and I wanted to see how the community is approaching this. Historically, setting up reliable ingestion meant writing custom PySpark streaming jobs, managing checkpoints manually, and dealing with schema evolution headaches when upstream source systems changed unexpectedly. Using Lakeflow Connect combined with Unity Catalog Volumes has noticeably streamlined how raw data hits the Bronze layer, especially for incremental file processing and maintaining clean lineage. I’m curious: Are you guys relying heavily on native Lakeflow/Auto Loader connectors for ingestion, or still building custom Spark/Delta streaming pipelines for specific edge cases? How are you handling schema drift when upstream source tables alter their structure? Any major pain points you've run into with CDC replication in Databricks so far? submitted by /u/SalamanderOpen859 [link] [comments]

00SalamanderOpen8594d ago
Databricks CommunityTechnical Blog

How to Query Unity Catalog Views from External Engines

005d ago
Databricks CommunityCommunity Articles

Genie agents and Unity Catalog wikis: what I tested, and what to use instead

006d ago
Databricks CommunityCommunity Articles

Unity Catalog secrets: what each grant does, and the schema-level READ SECRET that fails

006d ago
Reddit

Seeking feedback on my Databricks data engineering portfolio project

Hi everyone, I’m preparing to apply for junior/entry-level Data Engineer roles focused on Databricks, and I’d appreciate honest feedback on a portfolio project I’ve been building. It is a governed lakehouse project for a synthetic table-tennis e-commerce dataset: - Deterministic input contract and synthetic fixture with intentional duplicates, malformed records and quality issues. - Bronze ingestion designed around Databricks Asset Bundles, Auto Loader, append-only raw tables, lineage metadata and retained checkpoints. - Silver transformation with explicit validation rules and a quarantine layer. - Gold publication with privacy controls: only aggregated, non-PII cells that meet a k-anonymity threshold are published. - Unity Catalog / Terraform / dedicated service principal / least-privilege model. - Manual serverless Databricks Job with materialization followed by a read-only verifier. - 287 local tests covering contracts, transformations, bundle configuration, verifier logic and Terraform privilege rules. The current DEMO-001 cloud milestone produced: - Silver: 15 orders, 10 customers, 7 products - Quarantine: 6 expected rejected rows - Gold: €640 revenue, 5 orders, 0 refunded lines - Pre-suppression controls: €812.20, 13 orders, 2 refunded lines - Status: COMPLETED_WITH_QUARANTINE I have deliberately kept the scope honest: this is a fixed deterministic snapshot, not CDC, streaming, SCD, generic incremental processing, SQL Warehouse, Power BI, or a production monitoring setup. Repository: tt-ecommerce I would especially value feedback on: Does this demonstrate relevant Databricks/Data Engineering capability? What is missing before using it in applications? Would you prioritise finishing governance/consumer access, adding SQL Warehouse/BI, or CI/CD and operational monitoring? Is the documentation clear, or does it feel over-engineered for a portfolio project? What would you expect to see in a strong junior Databricks project that is not here yet? Thanks — direct criticism is welcome. submitted by /u/Alone-Cow-1563 [link] [comments]

00Alone-Cow-15636d ago
Reddit

Databricks Metric Views are coming back to Power BI! [Private Preview]

First I'd like to give a bit of background because if you feel like you've heard about this feature before, you're right. Previous attempt to bring Metrics Views to PBI was "blocked" when MS removed the required opiton from the connector. In February 2026 - the Power BI Databricks connector introduced Metric View BI Compatibility Mode (Beta). It allowed Power BI to query Unity Catalog Metric Views, with Databricks rewriting the generated SQL to handle measures correctly. In April , Microsoft removed that option from the connector. Reports relying on it stopped working. Now, in November , Databricks has announced a different approach. In new approach Databricks pushes metric view to Power BI semantic model, so ach metric view becomes a table, dimensions become columns, and measures are translated into DAX. You can publish the model from Catalog Explorer or automate it using a Lakeflow Job. There are a few limitations: - Synchronization is one-way, so changes made in Power BI don't flow back to Unity Catalog. - Import mode also changes the security model - imported data is governed by Power BI permissions. For me it's a long awaited feature. I've never been a fan of defining the same business metrics separately in different BI tools. In my opinion metric definitions should live as close to the data as possible, in a shared layer that different reporting tools can use. Connect metric views to Power BI - Azure Databricks | Microsoft Learn https://preview.redd.it/zysb9za0o51i1.png?width=1355&format=png&auto=webp&s=ff3ce75b6bd2c25735f1c368a0721f09cbed8627 submitted by /u/szymon_dybczak [link] [comments]

00szymon_dybczak6d ago
Databricks CommunityTechnical Blog

From AWS Glue Catalog to Databricks Unity Catalog: An End-to-End Migration Guide

001w ago
Reddit

Catalog Linked Database - Snowflake writes UC Managed Iceberg tables to Unity Catalog

Is Unity Catalog REST API only accessible via public internet from Snowflake when attempting to set-up a private link connection (catalog integration) to a Unity Catalog (Catalog Linked Database)? I need a private link connection between Databricks and Snowflake. The use case is Snowflake needs to be able to write Managed Iceberg tables to Unity Catalog. submitted by /u/RazzmatazzLiving1323 [link] [comments]

00RazzmatazzLiving13231w ago
Reddit

I tried to break my own open-source safety check for agent writes on Databricks: 5 SQL attacks, a 2M-row Delta table, 2 real bugs found

A pipeline can run green and still ship wrong data. With AI agents writing more of our code, I wanted a check before changes reach production. https://preview.redd.it/3w9kidmk8w0i1.png?width=1280&format=png&auto=webp&s=99f764348291e6a0c2a31dc1cb6d2c7028717153 bricksentry (free, open source) tells you three things before a change ships: • Who depends on this table (Unity Catalog lineage) • What the change does to the data (rows added, removed, duplicated, totals moved) • How to undo it (the exact Delta restore point) Created by Author. Created by Author. Then it says OK, warn, needs a human, or block. I tried to break it on a 2M-row table. Production was never touched, and I fixed three bugs along the way. Created by Author. Website and docs: bricksentry.org Code (Apache-2.0): github.com/bricksentry/bricksentry Package: pypi.org/project/bricksentry If you’re running agents against Databricks today, I’d like to hear how you handle change-safety, and what breaks when you run bricksentry on your own tables. submitted by /u/According-Future5536 [link] [comments]

00According-Future55361w ago
Databricks CommunityAdministration & Architectureanswered

Databricks Unity Catalog JDBC Connection: Can properties reference Databricks Secrets

001w ago
Reddit

Job completion triggers just hit Beta, and they replace a pattern I've built too many times!!

The usual way to chain jobs across teams has always been a bit ugly. Either a parent job with Run Job tasks that has to own everything downstream, or a schedule set 30 minutes after the upstream job usually finishes, which works until the day it doesn't. Databricks just added a job completion trigger in Beta. A job can now start when one or more upstream jobs finish with a matching result, and the release notes say those upstream jobs can live in other workspaces on the same Unity Catalog metastore. A few things from the docs worth knowing before you try it: - You can add a SQL condition, so the downstream job only runs when the data is actually ready, not just when the upstream job exited. - Unlike the other trigger types, it doesn't drop completions when the job is at max concurrency. It holds them until a run can start. - It's part of the multiple triggers Beta, so a job can have up to five triggers, and a workspace admin has to turn it on in Previews. Where I'd use it: a gold job that waits on silver jobs owned by another team, without either team having to own the other's orchestration. That handoff has been a friction point on every multi-team platform I've worked on. It's Beta, so I wouldn't move critical prod chains onto it yet. I'd start testing it in dev now though. submitted by /u/OkImprovement7010 [link] [comments]

00OkImprovement70101w ago
Databricks CommunityData Governanceanswered

Do Genie agents use Unity Catalog wikis as context?

001w ago
Reddit

Databricks Apps: Are you using them in production yet or keeping it to internal prototypes?

Hey everyone, I’ve been diving into Databricks Apps recently and wanted to see how others are adopting it in their organizations. Historically, whenever we wanted to spin up a Streamlit app or a custom Python dashboard connected to our data in Databricks, it meant dealing with a headache of external infrastructure—setting up App Service or EC2, dealing with network security groups, configuring service principals, and managing separate auth flows. Being able to run Streamlit/Dash apps directly inside the Databricks workspace, backed by Unity Catalog for governance and serverless compute, feels like a massive quality-of-life improvement for internal tooling. I’m curious to hear from anyone who has put them into practice: Are you using Databricks Apps for production-grade tools, or mostly for quick internal MVPs and demos? How are you managing your CI/CD deployments for these apps? Have you run into any weird gotchas with state management, scaling, or dependencies? submitted by /u/SalamanderOpen859 [link] [comments]

00SalamanderOpen8591w ago
Reddit

How much "ZeroOps" are you actually seeing in Databricks lately? (Serverless, DLT, Unity Catalog)

Hey everyone, I’ve been looking into the push toward "ZeroOps" within the Databricks ecosystem—specifically leaning heavily into serverless compute, Unity Catalog automation, and declarative tools like DLT/Lakeflow to reduce infrastructure maintenance. Marketing terms aside, I'm curious how much of this is translating to reality in production environments. For those of you managing large Databricks workspaces: Have you fully transitioned to serverless compute (SQL and notebooks), or are you still tied to classic clusters for specific workloads? How much time has Unity Catalog actually saved your team regarding access control and data governance overhead? Where does the "ZeroOps" promise break down for you in day-to-day operations? submitted by /u/SalamanderOpen859 [link] [comments]

00SalamanderOpen8591w ago
Reddit

Has anyone built production pipelines with Databricks Lakeflow Designer yet? Initial thoughts?

Hey everyone, I’ve been exploring Databricks Lakeflow Designer recently and wanted to see how the community feels about it. For context, it’s Databricks’ new visual, low-code authoring tool built on top of Delta Live Tables (DLT) for building ingestion and transformation pipelines. Historically, I’ve always written DLT pipelines purely in Python or SQL via notebooks or Git folders. When I first saw a visual designer, I was a bit skeptical (usually worried about "black box" code or messy JSON metadata under the hood). However, seeing that it actually compiles down to standard DLT/Spark and integrates natively with Unity Catalog lineage is pretty slick. I’m curious: Are you guys using Lakeflow Designer for production workloads, or keeping it strictly for quick ingestion and prototyping? How does it handle complex CI/CD workflows compared to traditional repo-based DLT code? Any gotchas or limitations you've run into with it so far? submitted by /u/SalamanderOpen859 [link] [comments]

00SalamanderOpen8591w ago
Reddit

Lakebase streaming sync is now GA

Hey Reddit, I’m happy to share one more new feature. The Lakebase streaming sync is now officially GA. Lots of customers are already using this feature to write streaming data with latency in 10s of milliseconds to Lakebase. It's much simpler to use than foreachBatch and you don't have to worry about connection pools, rate limits, error handling, etc. Here's what's new since we announced the Public Preview earlier this year: Declarative merge strategies*: Choose, per column, how an incoming row is reconciled with the existing row on a primary-key conflict. For example, you can keep the latest non-null value, keep the first value written, or take the value from the row with the latest event time. No custom SQL is required. External PostgreSQL via UC connections (Public Preview)*: Write to Postgres databases outside Lakebase, authenticated through a Unity Catalog connection so no credentials live in code. INTERVAL types*: DayTimeInterval and YearMonthInterval columns map to Postgres interval on DBR 19 and above. Serverless support: The sink now runs on serverless compute for notebooks and jobs, using the AvailableNow trigger. \ Merge strategies, external Postgres, and interval types require DBR 19.* Example: df.writeStream .format("postgresql") .outputMode("update") // optional; can be append mode as well .option("upsertkey", "primaryKey") // optional .option("upsert.columnStrategies", "last_updated_at:greatest,status:latestBy(last_updated_at)") .toTable("catalog.schema.table") SDP support remains in Private Preview. Public Preview, which adds support for native Lakebase streaming tables, is slated for later this year. submitted by /u/inkypinky [link] [comments]

00inkypinky1w ago
Reddit

How I check every prompt before the LLM on Databricks: GovTech's LionGuard on a small CPU endpoint

I wanted one shared safety check that every agent, job or SQL query on Databricks calls before the LLM. How it works: - LionGuard 2 Lite from GovTech Singapore: a small classifier on Google's EmbeddingGemma. It scores insults, hate, violence, self-harm and misconduct, and it handles English, Singlish, Chinese, Malay and Tamil. - Packaged as one MLflow model, registered in Unity Catalog, served on a small CPU endpoint. - From SQL, ai_query sends each question to the guard. Only questions with an unsafe score under 0.5 go to the LLM. Results on 7 synthetic questions: a Singlish insult, a Malay threat, a harmful Chinese message and an English "how do I hack the website" question were blocked. Safe questions in English, Singlish and Tamil were answered. Lessons: - The full set-up took 25 minutes. 19 of them were the endpoint start. - Do not use scale to zero for a production guard (cold start). - Turn on inference tables from day one, so you can show what was blocked. - It does not detect prompt injection. That needs a separate check. How do you check prompts for your agents today? Built-in AI Gateway guardrails, a custom endpoint, or both? submitted by /u/sundarleads [link] [comments]

00sundarleads1w ago
Reddit

[Beta] OneDrive connector in Lakeflow Connect now available!

Hey r/databricks , The Lakeflow Connect connector for Microsoft OneDrive (for business) is now available in Beta . You can ingest structured and unstructured files from a user's "My files" drive into Delta tables for analytics and AI workloads. Managed pipeline authoring is available through the Lakeflow Pipelines API or Declarative Automation Bundles (DABs) . UI-based pipeline authoring is not supported at this time, but will be available soon! Managed pipelines support incremental ingestion, automated retries, Unity Catalog governance, schema evolution for supported formats, and OneDrive-specific and standard file metadata. For direct control over ingestion logic, you can use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files , or COPY INTO pointed at OneDrive URLs. Common workloads Loading Excel, CSV, JSON, XML, Parquet, Avro, and ORC files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document for extraction, search, and agent workloads. Capturing file metadata without loading file contents. Examples using the Spark + SQL APIs Read an Excel range from OneDrive with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_onedrive_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load("https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents/Report.xlsx")) Incrementally ingest PDFs and Word documents, then parse them with ai_parse_document: CREATE OR REFRESH STREAMING TABLE onedrive_documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM onedrive_documents; CREATE OR REFRESH STREAMING TABLE onedrive_documents AS SELECT * FROM STREAM read_files( "https://mytenant-my.sharepoint.com/personal/user_mytenant_onmicrosoft_com/Documents", `databricks.connection` => "my_onedrive_conn", format => "binaryFile", pathGlobFilter => "*.{pdf,docx}"); Try the managed connector Ask a workspace admin to open Previews and enable Lakeflow Connect for OneDrive For Business . Create a Unity Catalog connection to OneDrive . Create a managed pipeline using Declarative Automation Bundles or the Lakeflow Pipelines API . The setup guide includes bundle YAML and an API example notebook . Schedule ingestion with Lakeflow Jobs. For DABs, define the scheduled job in the bundle. Current Beta scope The connector reads from a user's OneDrive for Business "My files" drive. For SharePoint sites, team sites, and shared document libraries, use the SharePoint connector . The connection authenticates as the signed-in Microsoft account and can ingest files that account can access. Managed ingestion requires Unity Catalog, Serverless compute, Databricks Runtime 19 or above, and the PREVIEW pipeline channel. DABs require Databricks CLI version 1.23.0 or above . Resources OneDrive managed connector documentation Managed pipeline setup: DABs and API examples Spark + SQL APIs and examples OneDrive connector limitations Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow If you try the Beta, please provide us your feedback! submitted by /u/BricksterJ [link] [comments]

00BricksterJ1w ago
Databricks CommunityData Governanceanswered

Unity Catalog secrets: GRANT READ SECRET ON SCHEMA fails with PRIVILEGE_NOT_APPLICABLE_TO_ENTITY

001w ago
Databricks CommunityData Engineering

Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation

002w ago
Reddit

Are people using Collibra to set owners of Tables in Unity Catalog?

Wondering if there's any value add of Collibra to Unity Catalog. Another tool to manage, wondering Databricks Request For Access is sufficient. How does your org manage access to datasets for technical and business users? Collibra or Databricks request for access or DABS? submitted by /u/RazzmatazzLiving1323 [link] [comments]

00RazzmatazzLiving13232w ago
Databricks CommunityCommunity Articles

Backing up Unity Catalog volumes without melting the driver

002w ago

Get Tuesday's version of this

Tracking Unity Catalog? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first