Service Principal
Recent items mentioning Service Principal across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks Java SDK v0.178.0 introduced dedicated fields to support PostgreSQL service principals and credential generation 3. In parallel, practitioner discussions highlight service principal management as a key focus when promoting Databricks Apps into production 4 and tracking deployment history within Databricks Asset Bundles (DABs) 5.
Generated daily from the 7 most recent items mentioning Service Principal. Click any [N] to jump to the source.
Governed Spreadsheets on Databricks: Row Zero Acquisition & What It Means for Business Users on Databricks (Demo + Chat)
Hey everyone! Had the wonderful opportunity of recording a demo + chatting with Breck, the founder & former CEO of Row Zero, a governed spreadsheets platform Databricks just acquired. For business users, this will pair up nicely alongside the Genie One / Genie Agents experience. Some quick answers to common questions the video answers (used AI + transcript for them): When did Databricks acquire Row Zero? =The acquisition was announced in September 24, 2026. Can Databricks customers use Row Zero today? =Yes, with a Row Zero account. A built-in Databricks experience doesn't have a date yet. How will it be priced? =Not final. Expected to follow the rest of Databricks (pay for what you use), not per seat. Does it respect Unity Catalog permissions? =Yes. Row Zero queries Databricks as the signed-in user (OAuth recommended, service principals supported), so row-level security carries through. Can you write back to Databricks? =Yes, to tables with a primary key. You review a diff before changes apply. Can you export back to Excel? =Yes, to Excel or CSV. Exports over about 1 million rows get truncated because of Excel's row limit, and admins can turn export off entirely. All roads in data begin and end with a spreadsheet after all! PS: Always love to hear your feedback on the video. submitted by /u/JosueBogran [link] [comments]
Seeking feedback on my Databricks data engineering portfolio project
Hi everyone, I’m preparing to apply for junior/entry-level Data Engineer roles focused on Databricks, and I’d appreciate honest feedback on a portfolio project I’ve been building. It is a governed lakehouse project for a synthetic table-tennis e-commerce dataset: - Deterministic input contract and synthetic fixture with intentional duplicates, malformed records and quality issues. - Bronze ingestion designed around Databricks Asset Bundles, Auto Loader, append-only raw tables, lineage metadata and retained checkpoints. - Silver transformation with explicit validation rules and a quarantine layer. - Gold publication with privacy controls: only aggregated, non-PII cells that meet a k-anonymity threshold are published. - Unity Catalog / Terraform / dedicated service principal / least-privilege model. - Manual serverless Databricks Job with materialization followed by a read-only verifier. - 287 local tests covering contracts, transformations, bundle configuration, verifier logic and Terraform privilege rules. The current DEMO-001 cloud milestone produced: - Silver: 15 orders, 10 customers, 7 products - Quarantine: 6 expected rejected rows - Gold: €640 revenue, 5 orders, 0 refunded lines - Pre-suppression controls: €812.20, 13 orders, 2 refunded lines - Status: COMPLETED_WITH_QUARANTINE I have deliberately kept the scope honest: this is a fixed deterministic snapshot, not CDC, streaming, SCD, generic incremental processing, SQL Warehouse, Power BI, or a production monitoring setup. Repository: tt-ecommerce I would especially value feedback on: Does this demonstrate relevant Databricks/Data Engineering capability? What is missing before using it in applications? Would you prioritise finishing governance/consumer access, adding SQL Warehouse/BI, or CI/CD and operational monitoring? Is the documentation clear, or does it feel over-engineered for a portfolio project? What would you expect to see in a strong junior Databricks project that is not here yet? Thanks — direct criticism is welcome. submitted by /u/Alone-Cow-1563 [link] [comments]
The notebookContent field for Clean Rooms asset notebooks is now optional instead of required. New fields have also been added to support PostgreSQL service principals and credential generation, as well as output type specifications for ML features.
Databricks Apps: Are you using them in production yet or keeping it to internal prototypes?
Hey everyone, I’ve been diving into Databricks Apps recently and wanted to see how others are adopting it in their organizations. Historically, whenever we wanted to spin up a Streamlit app or a custom Python dashboard connected to our data in Databricks, it meant dealing with a headache of external infrastructure—setting up App Service or EC2, dealing with network security groups, configuring service principals, and managing separate auth flows. Being able to run Streamlit/Dash apps directly inside the Databricks workspace, backed by Unity Catalog for governance and serverless compute, feels like a massive quality-of-life improvement for internal tooling. I’m curious to hear from anyone who has put them into practice: Are you using Databricks Apps for production-grade tools, or mostly for quick internal MVPs and demos? How are you managing your CI/CD deployments for these apps? Have you run into any weird gotchas with state management, scaling, or dependencies? submitted by /u/SalamanderOpen859 [link] [comments]
DABs deployment history
DABs can now record deployment versions, status, managed resources, and the user or service principal that performed each deployment. https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2 submitted by /u/hubert-dudek [link] [comments]
Databricks' UCX v0.61.0 makes UCX recognize more PySpark DataFrame write calls, including saveAsTable, when linting tables, and fixes pagination so large accounts get all account groups instead of only the first page. It also fixes API compatibility with DBR 16 in table migration workflows, upgrades Blueprint to 0.13.0, and moves the build from Hatch to uv.
What’s new in Genie Agents — October 2026
Databricks has introduced significant feature upgrades to Genie Agents, enhancing execution speed with Flash mode, expanding external tool actions via MCP, automating volume knowledge search, and introducing granular budget access controls. Agent Models and performance Flash (Beta) : Prioritize rapid execution speed over analytical depth for routine everyday tasks. Agent mode model upgrade : Agent mode now uses upgraded models that better handle long context and respond faster. Integrations and external tools Connect to external tools (Beta) : A Genie Agent can search and take actions in external tools like Google Drive through MCP connectors. Volumes and content search Faster file retrieval (Beta) : Content search parses and indexes your files so the Genie Agent can reason over a larger set of files, draw on specific passages across many files, and answer questions with lower latency. Content search details hidden from viewers (Beta) : The context panel no longer shows technical content search details, such as the number of files indexed, to users with CAN VIEW or CAN RUN permission on a Genie Agent. Automated Content Search and Volume Sync (Beta): When you attach a volume to a gent agent, Databricks automatically enables content search and keeps the index continuously updated as files are added or removed without inquiring any manual intervention. Volumes from restricted catalogs (Beta) : You can attach a volume as a source for a Genie Agent even when the volume’s catalog isn’t accessible from the agent’s workspace by default Authoring and Embedding Guided agent creation : A new guided experience walks you through creating an effective Genie Agent using Genie Code. It reads your data and suggests context that helps Genie answer questions accurately, such as descriptions of your tables and example queries. Deletion warning for sources and instructions : Deleting sources or instructions from a Genie Agent requires explicit permission. Volume file links in embedded agents : Links to volume files in an embedded Genie Agent open the file instead of redirecting users to the workspace. Governance and budget Budgets used to manage Genie costs support access control lists (ACLs) : Account admins and workspace admins can grant the Budget Manager or Budget Viewer role to specific users, groups, or Databricks service principals, delegating who can view and manage each budget. submitted by /u/Youssef_Mrini [link] [comments]
Power bi import gateway strategy
How are you guys that still use power bi via import mode designing your gateway authentication? We are new to databricks and have a lot of schemas that require different levels of AD access. Import mode is the company standard so no direct query sso auth. We are unsure how to design a scalable gateway strategy . Do we use a service principal or allow publishers to use their own credentials on the gateway? submitted by /u/curiouscsplayer [link] [comments]
How do you manage, in a Databricks Asset Bundles-based CI/CD, the permissions and access resources that a bundle cannot declare?
We have a DABs-based CI/CD in a monorepo and want to split it: deploy the DABs from the data-code repo and keep the Terraform infrastructure in a separate repo. The original idea was that Terraform would change at a different pace and its deployments would be manual, with no automated dependency on DABs deploys. However, at work they’re now saying this infrastructure part should be tied to the DABs deployment and not be manual, and I’m unsure where it should live and how to automate it. We mainly run into four cases: Cross-boundary grants: our pipeline needs to read a table owned by another team, so we need to grant SELECT on that table/schema to our service principal. Postgres grants: a job needs to connect to a Lakebase database and requires a GRANT on a Postgres schema, table, or role. Share contents: we need to add or remove tables and views from a Delta Sharing share consumed by an external consumer. Notification destinations: a job or pipeline needs to send alerts to a notification destination (email, Slack, webhook) that the bundle cannot declare. Today, all of these live in Terraform outside the bundle, so every time the data code needs one of these changes, we have to coordinate a separate MR and deployment in addition to the bundle deploy. Where do you put these in your pipeline: inside the bundle, in a separate Terraform stack applied after the bundle deploy, in the platform repo, or another way? What issues have you run into with each approach? Tx so much submitted by /u/Appropriate_Tip_9580 [link] [comments]
Run Lakeflow pipelines as a group: governance win or a way around SP gatekeeping?
As of this month you can set a Lakeflow pipeline’s run-as identity to an account group. Tables the pipeline publishes become group-owned in UC. Two patterns I expect to see: 1. Routing around SP provisioning. In a lot of orgs, creating a service principal is process-locked by a platform or identity team. Tickets, approvals, secret rotation. With run-as group, a team that already has an account group with the right grants can skip all of that. The conversation moves from “who can create an SP” to “can we just use the DE group?”. 2. Lower-maturity teams leapfrogging SPs. Teams still running pipelines as individual users will likely jump straight to groups. That’s better than personal identities, since nothing breaks when someone leaves. But it means they may never adopt SPs at all, and “good enough” becomes the permanent pattern. Questions for the sub: If you run pipelines as SPs today, are you planning to switch any to groups? Platform/admin folks: are you going to restrict this, and how? How are you handling auditability when the identity is a group instead of a single SP? Is group run-as a legit middle step for teams coming off user identities, or a dead end? My take: SPs still belong in prod. But I’d bet most orgs end up with group run-as spreading through both of these doors, and won’t notice until an audit asks who ran what. submitted by /u/OkImprovement7010 [link] [comments]
Databricks Genie One - Managing past conversation initiated by Databricks service principal account
Are PATs (user or service principal) accepted by Databricks Apps?
In a breaking change for AgentKit, the readMask field was replaced by a view field across managed memory entry requests. The update also adds OAuth scope support for service principal secrets, a deleted-deployment filter for bundles, and Slack and Teams destination settings.
In a breaking change, AgentKit managed memory entry requests replace the read_mask parameter with a new view field. The release also adds scope configuration for service principal OAuth secrets, deleted-item filtering for bundle deployments, and Slack and Teams destination settings.
Service principals created by Genie App builder are not visible in Databricks UI
The Terraform deployment engine for Databricks Asset Bundles has been removed, requiring users to migrate to the direct engine or remain on CLI v1.19.x. Additional updates include a fix for large integer corruption in API commands, group-level run-as support in bundles, and expanded authentication options for Databricks Apps.
You can now run an SDP pipeline as a group
A pipeline's run-as identity can now be an account group. Updates run with the group's permissions, and the tables the pipeline publishes are owned by the group. What it is Every pipeline runs as some identity. That identity decides what the pipeline can read and write, and who owns the tables it creates. Until now you could pick a user or a service principal. You can now pick an account group too. A Databricks role is a group under the hood, so setting run-as to a group also lets a pipeline run as a role. Why you'd want it If a pipeline runs as one person, it depends on that person's account staying active. When they leave and the account is deactivated, the pipeline can stop running and the tables it published can become inaccessible to everyone else. Service principals already fix part of this. Groups go a step further: a whole team owns the pipeline, and it keeps working as people join and leave the team. No owner reassignment when someone moves on. How to use it https://preview.redd.it/1ml56gev61uh1.png?width=1848&format=png&auto=webp&s=40275cf04ac3a837108289df8e8f16a855acfd98 In the UI: Open the pipeline, click Settings. Click the pencil next to Run as. Pick an account group you belong to, or one you have Assume permission on. Groups show up next to users and service principals. Click Save. The new identity applies on the next update. You can also assume a group’s identity to change / create a pipeline. And you can change the owner of a standalone MV/ST to a group as well! { "name": "sales-etl", "run_as": { "group_name": "sales-eng" } } Declarative Automation Bundles: resources: pipelines: sales_etl: name: sales-etl run_as: group_name: sales-eng What you need CAN_MANAGE on the pipeline. Either membership in the group, or Assume permission on it (for groups used for exclusive access). The group has to be an account group assigned to the workspace. Workspace-local groups don't work, and neither do the built-in admins, users and account users groups. Grant the group every permission the pipeline needs. Updates run as the group, so your own access to the source and data isn't enough. Docs: https://docs.databricks.com/aws/en/ldp/configure-pipeline#set-the-run-as-identity-to-a-group Please let us know what you think! submitted by /u/saad-the-engineer [link] [comments]
Azure Databricks, two-week-old account: partner pay-per-token endpoints (Claude, Grok) have never once succeeded — "Databricks-set rate limit of 0" — and the Assistant has "no daily token allowance". Open models work. What gates this?
Azure Databricks, Premium, one account with three workspaces, Unity Catalog, Azure Marketplace billing (no card, no trial credits left). The account was bootstrapped on a trial workspace about two weeks ago and moved to Premium six days later. I've spent two days on this and want a sanity check from anyone who has seen it. The facts, from system.serving.endpoint_usage: • Every open-weight pay-per-token endpoint has worked since the trial: gpt-oss-120b has ~50 successful calls going back to the trial period, Llama 3.3 70B likewise, zero refusals. • No partner pay-per-token endpoint has ever returned a success on this account. The very first call anyone made to databricks-claude-sonnet-5 was refused, and every call since, on every Claude endpoint and on Grok, in all three workspaces, for users and service principals alike: 403 PERMISSION_DENIED: The endpoint is temporarily disabled due to a Databricks-set rate limit of 0. • It is not our AI Gateway config: I removed every rate limit from one Claude endpoint and invoked it — same 403. The endpoints' config shows nothing else. • The notebook Assistant worked during the trial. On the paid account it refuses everyone, admins included: *"This workspace has no daily token allowance for the assistant."* Underneath, /ajax-api/2.0/conversation/llmproxy/ returns 429 {"type":"daily_token_limit_reached","daily_limit_tokens":0,"message":"…Daily token limit of 0 reached. Resets at midnight UTC. (DTB)"} and it recomputes to 0 every midnight. • Once, the Databricks Apps in the workspace were stopped by the platform with *"App compute was stopped due to workspace or account status"* while every workspace showed RUNNING. apps start brought them back. Red herring, for completeness: we also created a Unity AI Gateway per-user budget whose first version had a $0 threshold with BLOCK_USAGE. That blocked Genie for a day, exactly as documented, and was fixed. It was created a day *after* the first Claude refusal, so it isn't the cause of any of the above. What I've found: two Community threads describe "Databricks-set rate limit of 0" as a workspace trust-tier gate — trial-born accounts sit in TRIAL_VERIFIED, pay-per-token partner models are gated to PAYABLE_VERIFIED, a card alone doesn't flip it, only Databricks Sales/Support moves the tier. Both were AWS/personal accounts, neither resolved on-page. The account console's own API reports our account's feature_tier as STANDARD_W_SEC_TIER although the workspaces are Premium; I can't find what that field means. Questions • Does Azure Databricks with Marketplace billing go through the same PAYABLE_VERIFIED gate? Does it clear on its own after the first settled invoice, or does someone have to move it? • If you were moved: who did you contact (account team, or "Contact us" in the console) and how long did it take? • Is the Assistant's "daily token allowance" the same gate, or a trial allowance that simply goes to 0 when the trial ends on an unverified account? • Does feature_tier: STANDARD_W_SEC_TIER on the account object mean anything to anyone? submitted by /u/sumit671 [link] [comments]
Avoid Unity Catalog external location permission bugs by using managed storage
Hey, I work as a Databricks Engineer at Abilytics, so here's my take. If you are migrating existing tables to Unity Catalog, defining external locations incorrectly will break your grants. When you register an external location via CREATE EXTERNAL LOCATION, Unity Catalog validates your storage credential against the underlying cloud storage container. However, users often forget that read and write access on the external location is distinct from the IAM role or service principal trust policy. If a user has SELECT on a table, but lacks BROWSE on the external location, queries against parquet files directly via path-based access will fail with permission denied, even if the table-level grant succeeded. Always verify your storage credentials have explicit container-level policies attached before mapping external volumes. Furthermore, remember that path-based access like spark.read.load('s3://my-bucket/path') requires explicit external location grants, whereas managed tables abstract this entirely. Prefer managed tables default storage roots to bypass manual external location ACL synchronization issues entirely. Happy to go further — more on Databricks | Platform Engineering | AI at Abilytics , AI Systems, Data Engineering & Platform Engineering Services. submitted by /u/AbilyticsEng [link] [comments]
Databricks now deletes unused service principal secrets after 90 days
Lakebase Data API (GCP) returns jwk not found for valid service principal tokens
Issues with oidc/v1/token endpoint (REST for zerobus-ingest)
Anyone else work with the zerobus-ingest? There is an auth-flow to get an "oauth token" from a service principal with an "oauth secret". In that flow, you are supposed to submit the authorization details. See docs: https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-kafka I had been requesting access (via authorization_details) to a table as shown in the example: "type": "unity_catalog_privileges", "privileges": ["SELECT"], "object_type": "TABLE", "object_full_path": "main.default.air_quality" ... but it wouldn't work. Was banging my head on it all day. The privileges granted to the service principal for this table were unrestricted " ALL PRIVILEGES ". It turns out that, in addition to "ALL PRIVILEGES", this "token" endpoint wanted me to grant "SELECT " to the service principal in UC governance as well. Else the authorization_details are not valid. This "token" api has a lot of sharp edges. Any chance there is a nuget client to wrap around it and protect us from these unfortunate nuances? Is it documented that ALL PRIVILEGES doesn't actually give us all privileges, if we request "SELECT" by name? submitted by /u/SmallAd3697 [link] [comments]
Ontology ranked the snippet. I wrote the gate I wanted before trusting a Genie space.
A commercial user asked what yield was last season. Two definitions live in the same lakehouse. One is plot-level, moisture-adjusted, borders out. The other is as-harvested, and a dashboard still uses it. Genie returned the second number. Not a hallucination. OntoRank did what it does: creator, usage, link to a certified asset, freshness. The loud dashboard won. Nobody had certified the measure the board pack is allowed to quote. That is the failure I care about. A green demo is not a release gate. What I actually require before that space faces a user who will paste the sentence into Slack: Golden questions written before the space exists. If you write them after you have tuned instructions, you are scoring an overfit prompt. Benchmarks are not instructions. Genie does not learn from the benchmark SQL. The number lives in a Metric View. A column comment and a knowledge-store formula are hints. Ontology will rank all three. Only the Metric View is a contract every surface compiles. Same KPI from MEASURE(), from the certified dashboard, and from Genie. If they disagree, the space stays off Slack. Permissions checked as the asker. Inside a Genie Agent, compute is the author’s warehouse and data access is the end user. That split does not automatically hold in Teams, Copilot on maker credentials, or a Databricks App on its default service principal. The row filter still runs. It runs for whoever showed up on the SQL. A written kill line. I use Databricks’ “above 80% before UAT” as the hold line, not the ship line. Ship is 90% and zero fails on a regulated definition. Any miss on that definition takes the space down. A person does that. Not a service principal. Genie Code can draft the Metric View and the MERGE. ZeroOps can propose a fix and wait for Approve. Neither owns the definition. If the job stays green for eleven days because a key was slightly wrong, that is still your incident. I wrote the four pieces up so I would stop re-explaining this in reviews. Argue the 80 vs 90 line if you want. I will not move a regulated KPI to “looks fine in the demo.” submitted by /u/Worth_Phase1358 [link] [comments]
Databricks pipeline started failing after moving tables to Unity Catalog — how would you debug it?
Suppose a pipeline was previously using: hive_metastore.schema.table and is migrated to: catalog.schema.table The transformation logic hasn't changed. But the pipeline now fails during execution. Possible areas to investigate: Catalog permissions Schema permissions Service principal Pipeline identity Cluster access mode External location Storage credential Table ownership Hardcoded table references The interesting part is that the SQL may be perfectly valid. How would you troubleshoot this systematically? submitted by /u/hanshu6576 [link] [comments]
Databricks pipeline works in one workspace but fails in another — where would you start?
Imagine the same pipeline and code are deployed to two Databricks workspaces. Workspace A: catalog.schema.table → Query works → Pipeline works Workspace B: catalog.schema.table → Catalog not accessible → Pipeline fails The catalog exists, the table exists, and the SQL itself is valid. Would you first investigate: Workspace-to-catalog assignment Unity Catalog permissions USE CATALOG USE SCHEMA SELECT privileges Service principal permissions External locations Storage credentials Cluster access mode SQL Warehouse identity I'm curious about the actual troubleshooting order people follow for these kinds of Unity Catalog issues. What would you check first? submitted by /u/hanshu6576 [link] [comments]
"Do we need a metastore per workspace?" - the answer, and why the question keeps coming up
Every new joiner on our platform team has asked some version of this, so here is the short version. You do not. The Unity Catalog metastore is created in the account console and you have one per region, linked to as many workspaces in that region as you like. Dev, test and prod attach to the same one and share a single catalog.schema.table namespace. The mental model of "each workspace has its own metastore" comes from the legacy Hive metastore, and that is exactly what Unity Catalog replaces. What actually belongs where: - Account: billing, account admins, users, groups, service principals, and the metastore itself. - Workspace: notebooks, clusters, jobs, dashboards, the place people work. - Metastore: catalogs, schemas, tables, views, volumes, and the permissions on them. Identity federation means a group is configured once at account level and assigned to workspaces, so a grant on data applies wherever that group works. Workspace-local groups still exist in non-federated workspaces, but they cannot be granted Unity Catalog permissions, which is a good reason to migrate them. Storage: metastore-level managed storage is optional, and anything already in a bucket becomes an external location with a storage credential. Drew it out as a 2-3 minute animated diagram: https://youtu.be/2yy9-BL_RDo Series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4 How are you separating environments inside one regional metastore: catalog per environment, workspace-catalog bindings, or separate regions? submitted by /u/Broad-Cherry-5999 [link] [comments]
"Do we need a metastore per workspace?" - the answer, and why the question keeps coming up
This comes up constantly in real platform design, and the confusion is almost always about which layer owns what. The account is the container for the organisation, normally one per cloud provider. Billing, account admins and identities live there. A workspace is a single deployment of the UI: notebooks, clusters, jobs, dashboards. Most teams run dev, test and prod as separate workspaces. The Unity Catalog metastore is the part people place wrongly. It is created at account level, not inside a workspace, and the docs say you have one metastore per region, linked to any number of workspaces in that region. So dev, test and prod in the same region share one metastore and one catalog.schema.table namespace, instead of each keeping a separate legacy Hive metastore. Identities follow the same shape: users, groups and service principals are configured once in the account console and then assigned to workspaces, so you grant a group access to data once rather than repeating it per workspace. Storage is still yours. Metastore-level managed storage is optional, and existing buckets are registered as external locations backed by a storage credential. One line to remember it by: the account holds identities and billing, workspaces hold the tools and compute, the metastore holds the data and who can see it, shared across every workspace in the region. I drew it as a short animated diagram because the hierarchy sticks better visually (2:32): https://youtu.be/2yy9-BL_RDo Episode 3 of a series, in order here: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4 For those running dev and prod in the same region: are you isolating with separate catalogs, workspace bindings, or something else? submitted by /u/Broad-Cherry-5999 [link] [comments]
Can Databricks PAT authentication be used to mint an embed token?
Iam working on a Databricks integration where we support both PAT token and Service Principal authentication. The PAT connection works fine for making Databricks API calls, but when I tried to generate an embed token for dashboard embedding, it doesn’t seem to work. If PAT is not supported, is there any official reason or limitation behind this? Would appreciate any clarification from people who have implemented Databricks dashboard embedding. submitted by /u/Bhanuprakash_1947 [link] [comments]
What credential should I use to authenticate Databricks Zerobus?
I’m learning how to use Databricks Zerobus and I’m confused about authentication. When connecting to Zerobus, should I use a Databricks personal access token, OAuth credentials, a service principal, or another type of credential? I’m testing this for a learning project and want to follow the recommended secure approach. I won’t post any actual credentials. What authentication method are people using, and are there any setup steps or permissions I should know about? submitted by /u/Stu-dent999 [link] [comments]
Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Added data sources for managing external users, service principals, and groups at both account and workspace levels using v2 API versions. Introduced a new resource and data source for Postgres snapshot schedule configuration.
System generated Service Principal -App
Serverless Access Control is here!
We finally have a simple way to control access to serverless in the workspace. Two built-in objects have been introduced: - Default Interactive Compute - Default Automated Compute The first one supports notebooks and databricks connect. Second one jobs and SDP pipelines. To limit who can use serverless: Click Compute in the workspace sidebar. In the Serverless tab, click the kebab menu next to Default Automated Compute , then click Edit permissions . Remove the All Users group, or the group that includes all workspace users. Add only the specific users, groups, or service principals that you want to authorize. https://preview.redd.it/sg59jole9hkh1.jpg?width=957&format=pjpg&auto=webp&s=0c33d19d04c57856c51f57f33c44719317fe8347 submitted by /u/szymon_dybczak [link] [comments]
This release introduces v2 IAM resources and data sources at both account and workspace levels, covering management of groups, users, service principals, and workspace assignments. Documentation improvements include expanded Genie budget configuration guidance with shared versus per-user examples.
The SDK adds comprehensive IAM v2 API methods to both account and workspace clients for managing groups, users, service principals, and workspace assignments through create, read, list, and delete operations. The newCluster field in JobCluster is no longer required, making job cluster configuration more flexible.
The SDK adds new methods for managing groups, users, service principals, and workspace assignments at both account and workspace levels, providing expanded IAM capabilities. The NewCluster field in JobCluster is now optional, a breaking change that affects existing cluster configuration code.
The SDK adds comprehensive IAM v2 methods for managing users, groups, service principals, and workspace assignments at both account and workspace levels. The new_cluster field in JobCluster is no longer required, which is a breaking change.
The SDK adds job triggers for scheduling, API source connectors for pipelines, secret value retrieval in requests, new GpuXlarge8 workload types, Netsuite connection support, and token forwarding for apps. Breaking changes remove browse-related fields from model and MCP service APIs and make several user and service principal fields required.
The SDK removes the include_browse field from catalog service requests and browse_only field from service objects, which is a breaking change requiring code updates for affected users. New features include forward_user_access_token for applications, docker_image_url for AI runtime jobs, gpu_xlarge_8 GPU workload size, and required IAM fields for users and service principals.
The databricks_repo resource now supports an optional git_credential_id attribute to explicitly select credentials for repo operations. Workspace-level hosts now resolve workspace_id from host metadata instead of SCIM calls, preventing failures for service principals without /Me access and catching workspace_id mismatches at plan time rather than apply time.
The SDK adds support for AWS Bedrock access key authentication in the model provider service configuration. New Microsoft Entra service principal authentication fields are now available for Azure OpenAI and Microsoft Foundry provider configurations.
NewsWhy You Need Variable Groups in Azure Pipelines #AzureDevOps #DevOps #Shorts
Azure DevOps variable groups store environment-specific Databricks workspace hosts and authentication secrets. OAuth service principal client IDs and secrets enable secure connection to development and production workspaces within the pipelines.
Introduces catalogs.yml v2 support, a skip_optimize config for opting out of post-materialization OPTIMIZE, and Rust kernel backend for SQL warehouses. Fixes numerous incremental model bugs around constraints and tags, but now requires --full-refresh to apply changes to primary/foreign key expressions.
Declarative Asset Bundle Service Principals best practices
Zerobus ingestion fails in Databricks Free Edition using Service Principal OAuth
NewsDeploying Azure Databricks with Terraform? Watch this first!
This video demonstrates how to deploy an Azure Databricks workspace using Terraform by cloning a provided script, configuring variables, and executing Terraform commands. It walks through setting up prerequisites, authenticating Azure CLI, and populating a Terraform variables file to successfully provision the workspace.
The provider adds service principal Git credential management via principal_id on databricks_git_credential and enables permission management for Agent Bricks resources. Key fixes include metastore external_access_enabled now properly sent in PATCH requests, vector search index timeout increased to 75 minutes and made configurable, and workspace_id now accepting connection IDs alongside numeric workspace IDs.
NewsTerraform AWS Databricks Deployment Guide!
The video demonstrates how to deploy an AWS Databricks workspace using a provided Terraform script. It covers prerequisites, AWS and Databricks authentication, variable configuration, and executing the Terraform commands to create the workspace.
Why Databricks Genie returns MessageStatus.FAILED in App API (but works fine in UI) — solved
Hey fellow Databricks builders, Our team at **Enqurious** just wrapped a hackathon building a full medallion architecture (Bronze → Silver → Gold) with Genie AI integration for natural language querying. We hit a wall that cost us 6+ hours — sharing the full fix so you don't have to. # What was the setup? * Gold layer tables in Unity Catalog * Genie Space configured and working in UI * Databricks App deployed successfully # What causes MessageStatus.FAILED in Databricks Genie App API? The root cause is a missing service principal permission chain. The Databricks App runs under a **service principal** — not your user account. Genie working in the UI only proves *your user* has access, not the service principal the App uses at runtime. This is why the same question works in the Genie UI and fails via the App API every time. # What is the complete permission fix? The service principal needs **all five** of these — missing any one causes silent failures: sql -- 1. Catalog level (most commonly missed) GRANT USE CATALOG ON CATALOG your_catalog TO `your-app-service-principal`; -- 2. Schema level GRANT USE SCHEMA ON SCHEMA your_catalog.your_schema TO `your-app-service-principal`; -- 3. Table level — each table individually GRANT SELECT ON TABLE your_catalog.your_schema.table_1 TO `your-app-service-principal`; GRANT SELECT ON TABLE your_catalog.your_schema.table_2 TO `your-app-service-principal`; **Plus in the UI:** * SQL Warehouses → your warehouse → Permissions → Add service principal → **Can Use** * Genie Space → Share → Add service principal → **Can Run** # Where do you see the real error (not just MessageStatus.FAILED)? The actual error is in the **Genie Monitoring tab** inside your Genie Space — not the App logs. App logs only show the generic `MessageStatus.FAILED`. The Monitoring tab shows the specific SQL error or permission denial that caused it. This single tip would have saved us 3 of those 6 hours. # How long do Databricks permissions take to propagate? Unity Catalog permissions take **5–15 minutes** to propagate. Testing immediately after granting permissions is the most common reason the fix appears not to work. Wait 10 minutes, hard-refresh your browser, then retest. # Quick debug checklist If you're still hitting `MessageStatus.FAILED`: □ Check Genie Monitoring tab for the actual SQL error □ Confirm service principal has USE CATALOG (not just SELECT) □ Confirm SELECT on each table individually — schema-level alone is not enough □ Confirm SQL Warehouse is Running, not Stopped □ Wait 10 minutes after any permission change before retesting □ Hard refresh browser before retesting □ Verify app.yaml uses ["streamlit", "run", "app.py"] — not ["python", "app.py"] # FAQ **Q: Why does Genie work in the UI but fail in the Databricks App?** A: The App runs under a service principal, not your user account. The service principal needs its own separate permission chain regardless of your personal access. **Q: Is USE CATALOG required even if SELECT is granted on all tables?** A: Yes — USE CATALOG is mandatory. Without it the service principal cannot traverse the catalog hierarchy, even with table-level SELECT grants. **Q: Where is the real error when Genie fails in an App?** A: Genie Space → Monitoring tab. App logs only show the generic failure; Monitoring shows the specific cause. **Q: How long do Unity Catalog permissions take to propagate?** A: 5–15 minutes. Always wait before retesting. The Enqurious team wrote up a full walkthrough with SQL commands, screenshots, and the complete [app.py](http://app.py) template Happy to share our Genie Space config or [app.py](http://app.py) in the comments — just ask. **Questions for the community:** * Has anyone else faced similar Genie + App permission issues? * Is there a better way to debug these permission chains? * Would love to hear if Databricks is working on si […truncated]
This release fixes state-decoding failures in databricks_library, databricks_share, and databricks_quality_monitor resources introduced in the previous version. It also resolves workspace resolution errors affecting multiple account-level data sources and settings, including service principals and MWS resources.
Added Excel export format for assessment results and optional skip for workflow assessment tasks via configuration. Fixed multiple migration issues including federated catalog creation to include all external locations, external location name sanitization, GLUE credential lookup for external HMS, and improved error handling throughout the create-missing-principals and migrate-locations workflows.
Tutorials51 Setup Azure DevOps Pipeline with Databricks Asset Bundles (DABs) | Complete CICD Process
The video demonstrates how to set up an Azure DevOps pipeline to deploy Databricks Asset Bundles (DABs) to higher environments like QA. It covers configuring service principal permissions, setting up Azure pipeline variables for environment-specific details, and writing the YAML pipeline code to validate and deploy Databricks assets.
The assessment workflow now includes a force-refresh parameter to rerun assessments and obtain updated results, and a new experimental workflow converts Azure WASBS URLs to the more performant ABFSS format. Dashboard management is now more resilient, automatically recreating dashboards when permission issues occur or they're trashed, and service principals are now supported as members of account groups.
Assessment workflows require account groups to be created beforehand, and Service Principal is no longer supported for workspace installations—account-level installations now require Service Principal with Account Admin and Workspace Admin privileges. Table migration with default catalogs is fixed, workflow assessment filters to the last 30 days, migration progress workflows pause by default, and account group conflicts produce warnings instead of errors.
NewsDatabricks CI/CD: Azure DevOps Pipeline + DABs
Azure DevOps pipelines automate Databricks Asset Bundle deployments using service principals across dev, staging, and production environments. Configuration involves setting up service connections, variable groups, permission management, and triggering deployments from release and main branches.
NewsDatabricks As Code:Effectively Automate a Secure Lakehouse Using Terraform for Resource Provisioning
Rivian demonstrates how they automate over 95 percent of their databricks lakehouse using terraform to provision networking, workspaces, jobs, and secrets. The presentation also details their approach to managing unity catalog permissions, cluster policies, and cost control tags through infrastructure as code.
Get Tuesday's version of this
Tracking Service Principal? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.