Unity Catalog

Databricks Jobs to Kubernetes Spark Jobs migration

I'm discussing some architectural changes with my data team, and one of them is about cutting costs. The idea is to migrate Databricks Jobs to Spark jobs on Kubernetes. We've done a PoC and it works well. But despite the clear cost savings, I'm getting pushback from management and they're resisting the change. Can someone explain the reason behind this? It seems clear that you pay double for running jobs in Databricks, and you could save a lot of money. Yes, doing this means we bypass Unity Catalog, but we could convert only the Bronze and Silver layers. submitted by /u/worldpwn [link] [comments]

Databricks architecture drawn step by step: control plane, compute plane, Delta, Unity Catalog

Could a “Data → Agent” composer be useful for Databricks?

I've been thinking about a gap between Databricks data and agent frameworks. Databricks already has a lot of the building blocks: - Unity Catalog - Genie / Genie Agents - MCP - Vector Search - AI Gateway - Agent skills/tools - MLflow - Omnigent / Kasal And tools like Omnigent and Kasal already solve a lot of the agent orchestration/execution side. But I'm wondering about the step before that: What if a customer already has a large, curated and governed Databricks data estate — how do we turn that data estate into an agent-ready configuration without manually wiring everything together? Something like: Existing Databricks Data Estate ↓ Data-to-Agent Composer ↓ ┌────────┼─────────┐ ↓ ↓ ↓ Domains Semantics Metrics ↓ ↓ ↓ Genie MCP Skills └────────┼─────────┘ ↓ Agent Configuration ↓ Omnigent / Kasal ↓ Agent The idea wouldn't be to build another chatbot or another agent framework. It would be a Databricks-native composition/bootstrapping layer that understands an existing Unity Catalog/data estate and generates the pieces needed for agents to work with that data — domain boundaries, semantic context, approved tools, Genie configuration, MCP exposure, skills, policies, evaluation setup, etc. In other words: Kasal/Omnigent: Agent → Tools/Data Proposed layer: Data Estate → Agent I'm curious if this is already solved somewhere in the Databricks ecosystem, or if people are currently doing this manually when building enterprise data agents. Would love to hear how others are approaching the “existing data estate → production-ready data agent” problem. submitted by /u/imsuryya [link] [comments]

Feature enablement for Foundation Model Unity Catalog permissions

Built a Databricks medallion pipeline for NYC Taxi data

Been working on this as a way to get hands-on with Databricks Asset Bundles and Unity Catalog governance. It's a migration of an old on-prem NYC Taxi analytics stack (ClickHouse + Spark + Docker + Terraform) into a proper Bronze/Silver/Gold lakehouse. A few things I focused on: Auto Loader for incremental ingestion, triggered by file arrival Data quality handling that doesn't just drop bad rows — duplicates, zero-distance trips, and reversed fares go into dedicated quarantine tables instead of being silently discarded Databricks Asset Bundles for deploy/orchestration (dev + prod targets) 3 published AI/BI dashboards on top of the Gold layer (fleet ops, finance, compliance) It's intentionally small in scope — meant to demonstrate the lakehouse pattern, not be a production-scale platform. Currently only Green Taxi data; FHV comparison is planned next. Repo: https://github.com/Hamza-Bouali/NYC-DATABRICKS Would love feedback, especially on the Silver-layer data quality rules or the bundle structure open to critique. https://preview.redd.it/ej6hvzrbokph1.png?width=1667&format=png&auto=webp&s=b090b2c5165efcc83fb3f2b5c38ff6ef34c8ef48 https://preview.redd.it/rih9a5sbokph1.png?width=1879&format=png&auto=webp&s=1b85b90608e1b1273da4f8cbb4fa65d9f2c397a7 https://preview.redd.it/kjdor4sbokph1.png?width=1687&format=png&auto=webp&s=677c6194ede0cd01a8630ee98f85d11a5146b410 submitted by /u/No-Pollution-2274 [link] [comments]

Why Unity Catalog Managed Tables are recommended

Unity Catalog managed tables are the best choice you can make but do you know why? In the second episode of SuperSkills Oleksandra Bovkun and I demystify all the reasons to help you make your choice. Link to the video: https://youtu.be/Q7y8\_bSfVjQ submitted by /u/Youssef_Mrini [link] [comments]

Community BrickTalk | Real-Time Data & AI: Tripwise Demo

Hey r/Databricks ! Join us for community BrickTalk on Thursday, September 24 , focusing on real-time data streaming, AI agents, and governance using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Data and AI, giving customers a direct line to the people behind the products. In this session, we'll walk through a live demonstration of the Tripwise Demo , featuring: Sub-Second Transactions & Streaming: Device registration into Lakebase with sub-second reads/writes, plus telemetry streaming via Zerobus through a governed Medallion architecture. AI-Generated Offers & Pricing: Generating real-time agent offers using Foundation Model APIs and scoring behavioral data for usage-based renewal pricing. Natural Language Analytics: Enabling underwriters, product managers, and marketing teams to query governed insurance data in seconds using AI/BI Dashboards and Genie. Unified Governance: Managing safety, compliance, and control end-to-end with Unity Catalog. This is a great chance to see real-world architecture in action and ask questions directly to Databricks experts. When: Thursday, September 24 9:00 AM PT 12:00 PM ET 5:00 PM BST (London) 9:30 PM IST Register here and save your spot submitted by /u/Subject_Ant1789 [link] [comments]

Unity Catalog: Pros and Cons

Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began. In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems. That is where Unity Catalog comes in. submitted by /u/Low_Brilliant_2597 [link] [comments]

Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?

Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ⁠ai_query⁠ ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]

Non-deterministic ROW_NUMBER() results across Unity Catalog environments

Why External Secrets in Unity Catalog Matter

External secrets in Unity Catalog is in Beta, and it replaces Key Vault-backed secret scopes

This is the Databricks release I have been waiting for. Unity Catalog schemas can now hold external secrets, such as Azure Key Vault, and that will change how we manage and utilize secrets in our Databricks projects. On most of our engagements the secrets of record already live in Azure Key Vault, so we wire up a Key Vault-backed secret scope and move on. It works, but it is a workspace-level object from the pre-Unity Catalog era: configured per workspace, permissions managed through a separate secret ACL API, a flat scope/key namespace, and invisible to the governance model everything else on the platform now runs on. Read more: https://www.linkedin.com/posts/cenh_databricks-azure-unitycatalog-ugcPost-7504125176993800192-W9on/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]

Databricks Unity Catalog Explained | Full Governance Guide (Access Contr...

submitted by /u/macxima [link] [comments]

UC secrets in Key Vault

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC. more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Azure AI Foundry + Databricks Architecture | Deploy Genie Agent with DAB...

Azure AI Foundry Databricks architecture, Deploy Genie Agent with DABs, Databricks Genie Agent, Azure Databricks Genie Space, how to deploy genie agent with declarative automation bundles, azure ai foundry + databricks integration, fully operating genie architecture databricks, databricks unity catalog genie agent, azure databricks bronze silver gold architecture, agent to agent nlq databricks, databricks spark python sql delta lake unity catalog, production ready genie agent deployment, databricks vector search index genie, microsoft purview databricks governance submitted by /u/macxima [link] [comments]

Any plans to make externally backed secrets in Unity Catalog enter public preview/GA?

Hi Databricks Team, Seeking your advice on the above. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Actually understanding Unity Catalog Managed Tables

shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log

Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]

Unity Catalog Open Source in Name Only (UCOSINO)

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above). Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error. But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.) It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github! I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase. It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public. submitted by /u/SmallAd3697 [link] [comments]

How are you separating dev, staging and prod in Unity Catalog without duplicating everything?

Now that Unity Catalog manages Iceberg natively, is anyone actually switching?

Secrets in Unity Catalog

Secrets in Unity Catalog store credentials like API keys as governed objects named catalog.schema.secret: created, granted, rotated and audited with standard Unity Catalog privileges and redacted from notebook and job logs. This is a full end-to-end demo: 1)Create a secret with the REST API and in the Catalog UI 2)Read it in a notebook with dbutils.secrets.get 3)Use it to call the OpenAI API 4)Grant read/reference/write access to a user or group 5)Rotate it programmatically 6)Audit every access from a system table submitted by /u/Youssef_Mrini [link] [comments]

Community BrickTalk | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect

Hey r/Databricks ! We’re hosting a free, community-sponsored BrickTalk on Thursday, September 17, 2026, focusing on how to simplify and scale data ingestion using Lakeflow Connect! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights, giving you a direct line to the people building the products. Stop struggling with fragmented data across disparate sources. In this session, we'll demonstrate how Lakeflow Connect enables seamless data ingestion from SaaS apps, databases, and cloud storage directly into the Databricks Platform with zero infrastructure management. 🛠️ What We’ll Cover Native Data Ingestion: Learn how Lakeflow Connect provides fully managed ingestion directly into Unity Catalog as governed Delta tables. Simple Integration: See how to easily connect data sources using a simple UI or API. Accelerated AI & Analytics: Discover how unifying your data powers Customer 360, Operations, and downstream AI agent workloads. ⏱️ Global Times PT: 9:00 AM ET: 12:00 PM BST (London): 5:00 PM IST: 9:30 PM 👉 Register here to save your spot! submitted by /u/Subject_Ant1789 [link] [comments]

What’s new in Databricks - August 2026

Databricks shipped many major Generally Available features in August 2026. Here is the breakdown of what just landed: 🚀 Unity AI Gateway Enterprise AI governance layer covering model access, Model Context Protocol (MCP) management, and cost observability. 🔒 Role-Based Access Control (RBAC) Switch to scoped, temporary role assumptions instead of dealing with permission bloat. 🔑 Secrets in Unity Catalog Unified security secrets are now governed, 3-level namespace securable objects. ⚙️ Serverless Compute Access Control Granular admin controls over who can trigger serverless workloads across your organization. ⚡ Lakebase Postgres APIs & LTAP Direct Writes Accelerated synced-table loads and improved transactional data integration. 🤖 Genie Agent Upgrades Official GA releases for both the Agent mode API and Full-page Genie Code view. submitted by /u/Youssef_Mrini [link] [comments]

How to enable guardrails to an existing LangGraph agent deployed on a Databricks Serving Endpoint?

I have a LangGraph agent deployed as a Unity Catalog model on a Databricks Serving Endpoint. User : -> Databricks Serving Endpoint: -> UC Model: ai_workspace.agent.agent: -> LangGraph Agent: -> LLM + Tools The serving endpoint shows AI Gateway enabled , but its configuration currently only shows usage tracking. I would like to apply Unity AI Gateway guardrails to this agent. When I go to AI Gateway → Model Service → Create Model Service , I can configure guardrails, but my existing UC model: ' ai_workspace.agent.agent' does not appear in the Destination list. The available destinations appear to be Databricks-hosted foundation models and other supported model providers. My questions are: Can Unity AI Gateway guardrails be applied directly to an existing LangGraph agent deployed on a Databricks Serving Endpoint? If not, can I configure the LangGraph agent to use a Model Service as its underlying LLM endpoint, so that the LLM requests are governed by AI Gateway guardrails? Would those guardrails apply only to the LLM request/response, or can they also govern the LangGraph agent's tool calls and overall execution? What is the recommended Databricks architecture for applying AI Gateway guardrails to a custom LangGraph agent? Is the AI Gateway enabled option on a custom/agent serving endpoint different from the newer Unity AI Gateway Model Service functionality?

Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

Skills in Unity Catalog

Skills are available in Unity Catalog. They use a similar concept to volumes and are integrated with the AI gateway. New REST endpoints for skills are coming, and a new tool to manage them, ucode, is already available. more news: https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

CUSTOMER STORY | Kraken governs utility data at scale with Unity Catalog

Ingest image in databricks for powerbi ? a poc and any idea welcome

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security. That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table. What the simple poc below does: Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame. Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link. Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering: ​ Image_concat = IF( HASONEVALUE(images_in_delta[image_name]), CONCATENATEX( images_in_delta, images_in_delta[segment], , images_in_delta[split_index] ) ) Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement. The PySpark side, stripped to what matters — reading the images and doing the chunking: from pyspark.sql import functions as F from pyspark.sql import DataFrame #READ ALL THE IMAGES images_df = spark.read.format("binaryFile") \ .option("recursiveFileLookup", "true") \ .option("pathGlobFilter", "*.jpg") \ .load("/Volumes/main/image_ingest/image_sample") def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame: df_with_b64 = df.select( "*", F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url") ) df_with_split_info = df_with_b64.select( "*", F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"), F.length(F.col("base64url")).alias("total_length") ) df_split = ( df_with_split_info .withColumn( "split_index", F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1)) ) .select( "*", F.substring( F.col("base64url"), F.col("split_index") * max_len + 1, F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len) ).alias("segment") ) .drop("base64url", "content") ) return df_split def add_image_name(df : DataFrame) -> DataFrame : return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1")) df_images = add_base64url_from_image_binary(images_df) df_images = add_image_name(df_images) df_images.write.mode("overwrite").format("delta") \ .option("mergeSchema", "true") \ .saveAsTable("main.image_ingest.images_in_delta") Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally. At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking. Have you any other idea than this ? submitted by /u/Data-space_men [link] [comments]

Databricks Governance Hub! [BETA]

If you've been working with Databricks for quite some time you may have noticed that governance information is scattered across quite a few places. You might check Unity Catalog for one thing, system tables for another, admin pages for costs, and then somewhere else for tags or recommendations. The problem is getting a quick overall picture of the environment. The new Governance Hub seems like a perfect solution for above problems. It provides an account-level dashboard where you can get visibility into areas such as: How well data is being governed AI usage and related spending Where your Databricks costs are coming from Which resources have tags and how much tagged spend you have Potential governance gaps and recommended actions It's still in Beta, and there are some limitations, so I wouldn't treat it as the final governance solution. But for organizations running multiple workspaces, I can see this becoming a pretty useful operational view. Governance Hub - Azure Databricks | Microsoft Learn https://preview.redd.it/3y7jhevajvmh1.png?width=1720&format=png&auto=webp&s=8dd7324daa0e5e043634edc5633834ac91d4c117 submitted by /u/szymon_dybczak [link] [comments]

Unity Catalog - Can i delete system tags?

Databricks: 5 Minute Features - AI Functions

To celebrate that AI functions are now a fully governed part of Unity Catalog alongside the rest of your functions I thought it would be fun to do a run through of how easy it is to use. submitted by /u/Remarkable_Rock5474 [link] [comments]

An AI agent that validates and PRs your Unity Catalog access requests, and knows when to say no

Unity Catalog Schemas in Declarative Automation Bundles Are a Footgun

Iceberg vs Deltalake (greenfield project with UC in 2026)

I saw the quarterly meeting and was quite shocked that Iceberg is prominently mentioned, (as much as Deltalake). Is it possible that both are getting the same amount of love from Databricks? Is anyone aware of the R&D effort on these formats, and can share conclusions from that? If I'm building a greenfield Unity Catalog, should I just flip a coin to decide what format to use? Here are the main concerns and priorities: Which one is better for OSS Apache Spark reads and writes Which one integrates with external software better (eg onelake shortcuts pointing from Fabric to Databricks UC). When Databricks is innovating within their own UC (eg. introducing new managed table functionality such as "MST Transactions"), which one of these formats are they likely to support first? Which are they likely to optimize better? Which format is more likely to remain 100% open source in the future (or as close to open source as required by customers who want portable blob data). Sorry if this appears to be a common question. I am a Databricks outsider. I am more familiar with Microsoft Fabric. Where that Fabric SaaS is concerned, you can be certain that Deltalake receives a LOT more promotion than Iceberg does. We rarely come across Iceberg, and it probably wouldn't appear in any marketing slide decks. We are likely to create a gold/presentation layer in UC soon. It will basically be created from scratch. It would be nice to know which of these parquet-based formats to pick, when presented with the choice. I understand there is lip-service given to both, and it claims that this choice "doesn't matter". But that doesn't necessarily take into account the potential integrations that are needed with external software (eg. for the benefit of exposing the same tables in onelake). Is one safer than the other? Is one of them a better choice for forward-looking purposes? submitted by /u/SmallAd3697 [link] [comments]