What the community is asking.

Bypassing the Message Broker with Databricks Zerobus Ingest

Governing AI Strategy with Unity AI Gateway

Federation vs. ETL: Why I’m starting to hate moving data just to look at it.

I’ve been working heavily with Unity Catalog’s Query Federation lately, and it’s forcing me to rethink my "ELT everything" default. For a long time, the standard architecture was: Copy everything from operational DBs (SQL Server, Oracle, etc.) into the Lakehouse so we can join it. But the "ETL Tax"—the storage costs, the stale data, and the pipeline maintenance—is starting to feel like overkill for ad-hoc reporting. Why Query Federation is winning me over: Pushdown is clean: With Query Federation, Databricks pushes the SQL via JDBC to the source. The remote DB does the compute, and I just get the result back. It saves me from creating yet another massive ingestion job that I have to monitor. The "Governance" Win: This is the biggest selling point for me. I can manage security in Unity Catalog (UC) and it "wraps" the remote data. My users get the same catalog.schema.table experience, and I don’t have to manage 50 different connection strings for them. Automatic Lineage: Finally, I can see where data in a report actually came from, even if it originated in a legacy SQL Server. My Rule of Thumb (The "Architecture Bridge"): Use Federation for: Ad-hoc reporting, dashboards needing "live" data, and rapid data exploration. Stick to ETL for: Multi-terabyte aggregations, complex transformations (SCD Type 2, deduplication), and historical reporting. I’m not going to stress out a production operational DB with a massive GROUP BY at 10 AM. submitted by /u/SalamanderOpen859 [link] [comments]

Stop migrating data to Databricks: Are you using Lakehouse Federation yet?

submitted by /u/SalamanderOpen859 [link] [comments]

Stop migrating data to Databricks: Are you using Lakehouse Federation yet?

I’ve been diving deep into Unity Catalog lately, and the "Lakehouse Federation" feature is genuinely a game-changer for reducing ETL bloat. For a long time, the default answer to "how do I analyze this external data?" was "copy it into the lake." We’re finally moving away from that. Here is the breakdown for anyone trying to cut down on unnecessary data movement: The two ways to federate: Catalog Federation (The "Data-Centric" approach): How it works: Connects Databricks to external metadata (AWS Glue, Hive, Snowflake Iceberg). The Magic: Databricks compute does the heavy lifting (reading raw files in S3/ADLS). It’s fast, low-cost, and creates a "live" bridge to your storage. Best for: Incremental migrations and long-term hybrid storage. Query Federation (The "Database-Centric" approach): How it works: Pushes SQL via JDBC to remote databases (Postgres, MySQL, SQL Server, Synapse, etc.). The Magic: The remote DB does the compute. You get the result set back without touching the raw underlying tables. Best for: Ad-hoc reporting against operational/production databases where you don't want to break their performance with heavy ETL loads. Why it’s actually useful: Centralized Security: Unity Catalog acts as the single security guard. You don't need to manage separate credentials for your users on the remote DB. Automatic Lineage: This is the biggest win for me—Databricks automatically maps the lineage from the remote system to your report. No more "mystery data." Unified Syntax: You query everything using catalog.schema.table regardless of whether it’s sitting in Databricks, Snowflake, or an on-prem SQL Server. submitted by /u/SalamanderOpen859 [link] [comments]

Building Agents on Databricks with Custom Apps and Omnigent

Power BI to AI BI: The Migration Shortcut You Need

Solution Accelerator Series | Genome-Wide Association Studies

How to Connect Databricks and Fabric (No Copying Required)

Launch: Assury Cross platform agent lifecycle mangement

Hey HN, I am David Girvi, co-founder of Assury.ai. We are practitioners who became vendors last year. I wanted to share that we have launched agent lifecycle management for cross-platform agents. One centralized spot for all your agents. AWS, Databricks, Snowflake, Salesforce, etc. Lifecycle does auto-discovery in every environment, plus auto-tagging. We bring everything into a tamper-proof, searchable audit trail. Easily see who or what teams own the agents etc. We also have Kill/resume controls based on tagging so you can granularly shut down certain groups during an incident. We built this also to handle swarms, as one of our design partners has 30k agent swarms that spin up and down for research. Our thesis is that we want everyone to use the agents that will help them with their work. We want to be the observability, audit, governance, and incident response for all those different platforms. We started with one of the first MCP gateways that did execution level governance, then the endpoint with hooks, then our agent deception technology, and now lifecycle management. We are a small team of practitioners building product for our customers, not a marketing team. Cheers.

Databricks Ontos

Has anyone deployed Databricks Ontos? I don't see much information or demoes on this. https://community.databricks.com/t5/community-articles/semantic-bridge-to-unity-catalog/td-p/141416 https://github.com/databrickslabs/ontos https://preview.redd.it/uv9c00dvm8vh1.png?width=1718&format=png&auto=webp&s=293ab30f4ca38d135e9b08c6ecb332f617dda170 submitted by /u/manugp [link] [comments]

Announcement | Load terabytes of data in minutes into Lakebase Postgres

Minimize disruptions for continuous pipelines with Maintenance Windows

Serverless Job Compute vs Classic

We are a small Unicorn and have over 30 pipelines on a singular all-purpose cluster compute. Now I am not so sure which job compute is better? / for what usecase do they differ? submitted by /u/steroids-bad [link] [comments]

Disaster Recovery

Hi all. I saw Databricks managed recovery released which could be good but still has a lot of limitations. In the meantime how are people managing DR, especially around data in managed tables? If we create external tables then recovery of azure storage and then pointing new tables in UC to restored storage is easy. But managed tables with the guid folder structure is harder. Ideas I had were to find a way to script out tables and locations that runs regularly, and then a DR script that could use that as an input, so when you have a DR situation and you need to stand your environment back up again, it could create the new managed tables and then populate them by looking at the right folder in your restored storage. Anyone else have some good solutions around this? submitted by /u/angryapathetic [link] [comments]

Account Registration Help for Partner-Academy

How to automatically generate Spark schemas from XSD files

I have xml files and associated xsd file so I want to read the xml data based on xsd schema. submitted by /u/BugSquare4344 [link] [comments]

Databricks - SQL Incremental Load Question

Hey guys I'm relatively new to Databricks but want to lead the charge to move to it in my company instead of just using ADF for everything. In regards to Delta loads (loading data incrementally based on when it was last updated) what I would originally do is run a BINARY_CHECKSUM() on all columns and then track what has changed before bringing in the differences. This however is not supported in databricks and I want to do things properly so how would you do this? Source Data (Azure SQL) Bronze Layer (Azure SQL) What is different between them? What new rows have been inserted? What rows in Bronze no longer appear in Source What I love about Databricks is that you can do joins of different data sets within the same activity in a notebook but when I tried to do this with manually comparing every single column, it takes forever to load. I just want to know the best way of doing this so any help or resources would be appreciated. Many Thanks! submitted by /u/RevolutionaryRich882 [link] [comments]

issues creating a foreign catalog

Are PATs (user or service principal) accepted by Databricks Apps?

Career Advice in Databricks

Hello, Kindly take a movement to read below message. I am sap technical data migration professional , planning for my career transition to Databricks. could you please advice on the best learning path and roadmap for cert". Specifically, I would like to understand: Which Databricks certif" should I pursue first? What learning path would you recommend for an SAP technical professional? Which skills and technologies should I focus on to improve my career prospects in the Databricks ecosystem? I look forward to your valuable insights and recommendations. Thank you. submitted by /u/alive_at25 [link] [comments]

Best Practices for Managing Permissions in Databricks Workspaces

Managerment resource computer

Building an Expense Management App with the Genie App Builder Leveraging Lakebase + AI for Image Scanning

I previously did not consider Databricks a great place to build apps on with AI. With the release of Genie App Builder (still in beta), that has changed, and now I see it as a serious player in the AI app building space. Before I start, quick disclaimer: I am part of the Databricks MVPs, work for a consulting company that is a Databricks partner, etc. That said, this review is fully based on my experience, my own words, my own testing, etc. Always try things out yourself! Things I appreciated: +By far, this feels like the most polished Databricks beta that has been released. If the goal is to make this easy for business users to develop applications easily and securely, this feels like it is already there. Granted, I have not tested with multi-users yet, but it feels very polished already. +Native integration with Databricks features like Lakebase. +Even though it has native integrations to Databricks features, it didn't force these features in building what I did when it was otherwise unnecessary. For example, the dashboard it built as part of the application was not an AI/BI Dashboard. I would have expected it do that, but it didn't, and rightfully so, as it would not have been truly the best thing for this use case, so truly a customer first thing here. +Respects governances/access controls, meaning I can develop an application freely in the sandbox environment, but if I want to deploy it, organizational safeguards, Unity Catalog, and other controls built on Databricks apply. Things that could be better: -With the goal of empowering citizen devs (or i should just say, business users that are technical enough), there needs to be an easier way to send request for access/capability etc to admins. Right now, the builder tells you through the chat, but that is not as intuitive as having a clear UI-embedded to-do things before deploying etc. One step further would be that the developer can notify the administrator of this need. -I found that when I developed an application that had to use a sandbox only capability when access was otherwise unavailable (see above), just hung when I tried to reopen the dev experience for it. I had to troubleshoot through the chat interface as to why that was happening to figure this out. Additional Note : Currently free at the moment, but pricing is not yet available since we are in beta. Will be interesting to see how the overall value of this capabilities plays out factoring the ease & governance vs compute & token costs. Closing Thoughts I think this is on track to be one of Databricks' best released feature, and a serious entry in the space that is otherwise feel with undifferentiated app building solutions. Is it creating a new category? No. But it feels like an Apple-level product, where being the best at executing is sometimes better than being the first to do something. And I believe it will pay off as businesses look more carefully as to where they spend their hard-earned money on. From a "data + AI platform vendor" perspective, this is currently the best offering of its kind. From an "app building vendor" perspective, there are still solid players out there, but for enterprise focused apps, in the top 3 of options I would consider, and for sure the first one I would try if my stack was heavily Databricks focused. submitted by /u/JosueBogran [link] [comments]

How would you detect changes in thousands of provider XML feeds without treating every job as new?

submitted by /u/Mediocre-Goose-380 [link] [comments]

Your pipeline works perfectly in dev. Then production happens. What would you check first?

submitted by /u/iamadeculprit [link] [comments]

Things I miss on Databricks

Here are a couple of things I miss on Databricks: Group jobs per category. We all know that jobs can become messy sometimes. Connect jobs in Git more easily Data Quality. Like for real, where users can establish rules, create scans, quarantine, break pipelines, and etc. I don’t want to take my data outside to do it. App Genie Builder that won’t force me going to bundle Scale to zero apps without needing to use App Genie Builder Similar to 1, grouping queries, we all know that it can get messy with time Auto TTL in Lakebase (don’t know if it’s possible, but that’s a very good functionality) What else do you miss in Databricks? submitted by /u/Select_Morning8221 [link] [comments]

Genie spaces with DABs - how do you deal with the table ordering?

Nanosecond timestamps

Databricks added nanoseconds precision TIMESTAMP(9) and TIMESTAMP_NTZ(9). TIMESTAMP(9) 09:30:00.123456789 would be 09:30:00.123456 as ordinary TIMESTAMP. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Govern Models and Model Services Using ABAC Grant Policies

Databricks Acquires Responsive

Responsive is joining Databricks to accelerate development of Lakebase. submitted by /u/sai-nageshwaran [link] [comments]

Responsive Is Joining Databricks

--- top comments --- [andrenotgiant] The "database on object storage" gang is really coming together.

Building on Lakebase | Episode 1: Meet Lakebase

Lakeflow Connect SQL Server (integrated CDC pipeline): no way to Include/exclude columns

Building an End-to-End FinOps Cost Attribution Model for Azure Databricks

Databricks AI Platform Kit - Quickstart

Your Ontology Records Permission. It Does Not Resolve Authority.

Agents can read your policies. They can't read the room. Formal authority is easy to model: the org chart, the approval process. But a lot of real decisions run on informal authority that's never written down, whose sign-off actually matters, whose pushback kills a project. A human picks that up in their first month. An agent never does. It trusts the documents completely and can do everything "right" on paper while still getting it wrong. Read this piece from Parviz Shariff Principal EA at Coca-Cola HBC arguing that before an agent acts on an approval, it should check whether that approval matches how the decision actually gets made in practice. From an ontology angle, how would you even represent "whose sign-off really matters here"? Model it explicitly, mine it from behavior, or treat it as out of scope? submitted by /u/Berserk_l_ [link] [comments]

Community Bricktalk | SQL-Native Vector Search

Genie "Field" error: Table name or alias is required for column 'amount'

Databricks on GCP - Setting Up Backend PSC for your Workspace

Create a Classic Databricks Workspace using AWS Automated Configuration

Any Metadata extract and backup utility for UC?

Does anyone know of a github with the ability to connect to UC catalogs via SDK and extract all the metadata for backup purposes? Such a thing would be helpful to have in source control, in text format. It would give a high-level representation of the catalog that could be monitored to see changes over time (when tracked in a git repo) I was happy to find this one: https://github.com/databricks-solutions/databricks-replicator/tree/main ... although it has a very unfortunate license. I'm looking for something that is at least as permissive as the "reference implementation" of UC catalog itself, also in github Apache License 2.0 https://github.com/unitycatalog/unitycatalog/blob/main/LICENSE submitted by /u/SmallAd3697 [link] [comments]

Supervisor Agent Deployments!

Monitoring a custom Agent (Responses) serving endpoint with an experiment on the Unity Catalog

Single DAB covers your entire agentic app architecture on Databricks

"Your request was rejected since your organization has exceeded the rate limit"

CUSTOMER STORY | CARIAD unifies global automotive data with OpenSharing and Unity Catalog

Cohort size of databricks APM?

Hey does anyone know the estimated cohort size for databricks apm fulltime? submitted by /u/Worried-Space-6545 [link] [comments]

Need Buddy

Buddy for Databricks Event going in Bangalore at 15 Oct . submitted by /u/SafePack7665 [link] [comments]

Data Starts AI, Context ships it to production. Must watch video for data professionals and leaders.

submitted by /u/ConstantNo2668 [link] [comments]

Databricks Genie App Builder is now in Beta (you can turn it on in Previews)

submitted by /u/CautiousUse8597 [link] [comments]

Liquid Clustering vs partitioning + Z-Order: has anyone fully switched, and when does the old approach still win?

Databricks now recommends Liquid Clustering for new tables, but a lot of existing pipelines still use partitioning + Z-Order. For those who’ve migrated or tested both: Did you see real query or write improvements, or was it a wash? Are there cases where you kept partitioning on purpose (huge tables, partition-level overwrites, retention by date)? How did you pick clustering keys, and did you ever have to change them? Curious what’s actually working in production vs what the docs say. submitted by /u/UnluckySugar610 [link] [comments]

Is Delta Sharing the end of external ETL? My notes on Open vs D2D.

Just spent some time digging into the Unity Catalog and Delta Sharing documentation. Here’s the TL;DR on the two patterns for anyone looking to implement this: Open Sharing (The "Universal" Path): Audience: Recipients without Databricks (using Power BI, Pandas, Tableau). Mechanism: Presigned URLs + Bearer Tokens. Constraint: Tabular data only (tables/views). Security: Can be restricted via IP filtering. D2D (The "Native" Path): Audience: Both parties are on Unity Catalog-enabled Databricks. Assets: Tables, Views, Volumes, Notebooks, and Models. Pros: No token management; Zero egress costs if in the same region. Performance: DBR 16.2+ uses scoped credentials, making it basically as fast as a local table read. Observations: The "Zero-Copy" aspect is the real winner here. The audit trail in system tables alone covers about 95% of standard compliance needs (GDPR/SOC). The Coastal Community Bank case study (2 months to 2 days for onboarding) seems to be the gold standard for ROI here. Discussion: Is anyone actually using the Open Sharing side with external vendors in production? How are you handling the token rotation/security for non-Databricks clients? Or are you finding that most of your partners are already on the Databricks ecosystem? submitted by /u/SalamanderOpen859 [link] [comments]

How to transfer Databricks certificates and badges when changing employers

We Built Seemore for Databricks

Databricks cost optimization: monitoring all-purpose clusters with the API

Databricks costs directly within the interface – FinOpsWay extension

The Breakdown: Databricks

submitted by /u/Roadtochessmaster [link] [comments]

Airflow said success. Finance said the numbers were wrong. Finance was right.

submitted by /u/IamculpritforDE [link] [comments]

Databricks ABAC Now Follows Your Data: External Engines and Views

Databricks ABAC row filters and column masks now reach external Apache Spark (GA) and Unity Catalog views (Beta) submitted by /u/Lenkz [link] [comments]

Insurers Don’t Need Faster APIs. They Need Fewer Calls: Geospatial Scoring with H3 and Spatial SQL

I thought Databricks Lakebase was expensive

I used to think Databricks Lakebase was pretty expensive. A few Databricks architects I've talked to thought the same, especially when you factor in syncing data between Lakebase and Lakehouse. With LTAP coming, that second part should become less of an issue. Recently, we were adding Lakebase pricing to our free FinOpsWay extension so you can see estimated costs directly in the Databricks UI when creating a database. And I was honestly surprised by the numbers. Compute was cheaper than I expected, especially with scale-to-zero. Obviously, it depends on the workload, but I think I dismissed Lakebase too quickly without actually looking at the pricing. Anyone running Lakebase in production? Curious how much you're actually spending and whether it's worth it. submitted by /u/Significant-Guest-14 [link] [comments]

Anyone tried generating dashboards with Genie using complex prompts?

Like if the same business term means different things to different teams, how do you make sure Genie understands what you actually mean!!.. Do table and column descriptions do the job, or do you need to add example queries and instructions too I tried creating dashboards using prompts with different scenarios, like SQL datasets with calculated columns, 1 million rows, different numbers of widgets, multiple datasets, parameters, and global filters. Most of the results were correct, but when I tried the scenario of creating a dataset from views, the generated dashboard had no preview data. So I amm wondering how much the context and instructions actually matter here. Has anyone faced similar issues with Genie, especially when working with views? submitted by /u/Bhanuprakash_1947 [link] [comments]

What happens when too many ppl run queries on the same Databricks SQL Warehouse?

Do the queries just wait in a queue, or does the warehouse automatically handle the extra load.. Also, how do you figure out when it's time to increase the warehouse size instead of letting queries wait... Curious how this works in real projects. submitted by /u/Bhanuprakash_1947 [link] [comments]

ServiceNow ingestion failed since October 1st.

Use multiple triggers for Lakeflow Jobs

A Lakeflow Job can now have up to five independent triggers. Any active trigger can start the job. For table updates, specify whether all or any monitored table must change; in DABs, use ALL_UPDATED or ANY_UPDATED. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Declarative Pipeline with editable wheel: PYTHON.MODULE_NOT_FOUND_ERROR

Insurers score millions of locations through an API. The API isn't slow, the unit of work is wrong.

Multiple triggers on one Lakeflow job (beta): finally killing the duplicate “month-end close” job

I can't tell you how many times I've heard this common finance ask: refresh reporting tables daily at 6 AM, but during month-end close, run every hour plus whenever the ERP drops a reconciliation batch. Until now that meant multiple jobs pointing at the same notebooks (or a parent job with run-job tasks), and keeping parameters and dependencies in sync across both. There’s now a beta feature that allows multiple triggers on a single job: Enabled per workspace by an admin under Previews (“Multiple Triggers”) Up to 5 triggers per job during preview Mix types: multiple crons, table update, file arrival Each trigger pauses/resumes independently In DABs / Jobs API you define a list of triggers instead of the single schedule block Gotcha: any trigger firing starts the job, and max concurrent runs defaults to 1. If a file-arrival run is still going when the cron fires, the second run (queues/is skipped, depending on your queue setting). Worth testing the overlap before you put this on anything high priority. Has anyone tried this yet? Curious how others are finding uses for this. submitted by /u/OkImprovement7010 [link] [comments]

TellR Roadmap and Strategic Direction | @Databricks Team

Certificate for Databricks course

Revive! Metabolism: A Practical Approach to Everyday Wellness

Databricks Dataset Materialization

Letting devs use any AI coding agent against governed data, without keys on laptops. What actually works!!!

Disclosure up front: I'm a forward-deployed engineer at Databricks, so I work on this, but this is a hands-on write-up, not a pitch, and the pattern's the interesting part. The problem I keep hitting at enterprises: devs already use a mix of AI coding agents (Claude Code, Codex, Cursor, Copilot). Leadership wants to "standardize on one," which fights reality. The real question isn't which agent, it's how to let all of them touch enterprise data safely. I spent a week wiring agents through a governed gateway + MCP and trying to break it. What held up: - per-user/per-model entitlement enforced server-side (not a client checkbox) - governed data tools that are fail-closed. no valid identity, no access - reads scoped to the user's catalog permissions. the agent sees only what the human can - human-in-the-loop by default (shows the query before running, no silent writes) Write-up: https://wesdias.medium.com/how-to-connect-any-ai-assistant-to-governed-enterprise-data-with-mcp-0d9cffa14231 How are you handling this? governing agent data access at the gateway/MCP layer, or still per-tool API keys and hope? What's fallen over for you? submitted by /u/wesleydias84 [link] [comments]

Run a Databricks job when a file on git / DABs changes

DABs can run a job during deployment when a tracked file changes (JSON, CSV, etc.). Use job_runs with on_file_change to refresh configuration or merge it into a Delta table. whole article: https://medium.com/databrickscommunity/databricks-dabs-ci-cd-4-ways-to-automate-your-deployments-6a732d9e928c https://www.sunnydata.ai/blog/databricks-asset-bundles-job-runs submitted by /u/hubert-dudek [link] [comments]

readme in the external locations that are hosting catalogs (unity storage)

Is there a common convention used to annotate the ADLS locations where unity catalogs drop their files? When browsing storage to these locations using ADLS tooling, we are finding a messy assortment of meaningless guids and parquet files. There is little that is placed in here to make sense of the blobs. It seems doubtful that we are intended to memorize these guids: __unitystorage/catalogs/123453e0-6655-448c-b053-1234560fb834/ tables / and ../ volumes I don't know why it would be so hard for databricks to drop an " index.htm" file or something, and keep it updated whenever new tables are added or removed from the managed catalog. At a very MINIMUM, such a file could display the workspace bindings and catalog name since these change very infrequently. I'm guessing that if I want to have an index.htm file then I'll have to do it myself, and probably put it under " volumes ". Does anyone else do this? Is there an existing convention or utility to emit such a file in a standardized way? submitted by /u/SmallAd3697 [link] [comments]

Deploy via Databricks Express Setup

Databricks AWS Workspace Manual Deployment

Get the 50% voucher after completing the Databricks Modules for PDE Cert

AI/BI + Genie Launchpad (Oct 20–22) | Free Hands-On Training

Personalizing Genie Code with MCPs and User Skills

Beyond Bronze, Silver, and Gold: When Does a Separate Stage Make Sense?

Finally, a Simple Way to Deploy Databricks Hybrid Workspaces

Unable to Link Azure Account to Databricks

Build a Document Knowledge Base on Databricks

Locking the folder does not lock the answer. On Databricks you can grant the volume, the table, and the search index. None of those grants decides which page a person may read. The index does not apply the row filter. Someone who can query it gets whatever the question asks for.The label on each chunk has to be set by your own service. The person asking does not get to choose it. submitted by /u/Worth_Phase1358 [link] [comments]

Databricks and SAP: Building the Next Generation of Enterprise Data and Analytics

Propagating SCD2 History from Silver to Gold with AUTO CDC FLOW and Handling Full Refresh

How do I create unit tests with SDP Pipelines?

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case: Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right? But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing. With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors Checkout the blog for a deep dive tutorial on how to use unit testing with SDP Pipelines! submitted by /u/data-baggins [link] [comments]

Solutions Architects (Or SEs) Stock and Comission

Hey folks, without getting into specifics. Do SAs/SEs typically hit their commission numbers? Would anyone provide a range of expected quota, without violating any PII. IE, is it 5m a year, 50m a year, etc. Does DB typically do stock refreshes and typical raises? What is the effort/process to go from L4 to L5 (outside of typical requirements). Is it done pretty often or rarely? Just trying to get a feel for the business, as I have heard that many positions start very low in the business and I have to determine what/if it is viable for my family. Thank you submitted by /u/konotiRedHand [link] [comments]

Question on offer

submitted by /u/AccomplishedTeam5008 [link] [comments]

How to set budgets and alerts on SQL warehouses so self-service never surprises Finance

Domains and Subdomain tags are not key/value pairs

We are actively working on improving the everything related to Genie Ontology. An annoying thing that we found is that when we create domains, we can only assign bare keys and not key/value pairs? This means that we will have a massive list of domains and subdomains tags in the "governed tags" list instead of 2. Is that really how it's supposed to work? submitted by /u/molkke [link] [comments]

What's new in Databricks AI/BI (Oct 2026): multi-fact modeling, Row Zero spreadsheets, Excel Add-in GA

If you work in BI on Databricks, this monthly roundup might be useful. This month's edition covers: Relationships now support multi-table schemas and measures across fact tables with different granularities (via shared dimensions) "Ignore filters" on dashboard widgets, handy for calendar hierarchies Row Zero acquisition: spreadsheets on billions of rows with a live connection to governed data Excel Add-in is GA: browse Unity Catalog, pull metric views in as native pivot tables, and ask Genie One from inside Excel Full post: https://aibilakehouse.substack.com/p/beyond-the-star-schema-spreadsheets Disclosure: all three authors work at Databricks. Happy to answer questions or hear what you'd like covered next. submitted by /u/Good_Robin [link] [comments]

Bricktalk Recording | Databricks Certification - Did You Know?, What's New, and Where We're Going

The next gen of data infra optimization is about connecting cost to value

--- top comments --- [austin-cheney] I don’t have experience with Databricks specifically but I will be basing my startup on the idea that with superior realignment of memory management I can improve data transmission performance 40x compared to incumbent solutions, or 10,000x if weening businesses off http, and then transferring those performance improvements directly to lower infra costs. The goal is to come in with superior technology and charge a tenth of the competition. I can still kill on margin and everyone is happy just on the numbers.

Announcement | Meta’s ads MCP server comes to Databricks

When a Pyspark job really slow, what's the first thing you check?

I've been curious about how people actually troubleshoot slow Spark jobs in real projects. There are so many things that could be causing it,like shuffles,data skew,joins,partitions, or small files. Do you usually start with Spark UI, check the query plan,or look at something else first? Would be interesting to hear how people approach this, especially when a job that used to run fine suddenly starts taking hours. submitted by /u/Square-Designer7807 [link] [comments]

Service principals created by Genie App builder are not visible in Databricks UI

Add custom_tags and budget_policy_id support to Declarative Automation Bundle presets

Unable to Serve Gemini 3.8 Flash on Azure Databricks – East US

Run Google Dataproc (Managed Spark) on local, for offline dev, CI, and AI agent testing (multi-arch, zero cloud cost)

may of us, have multiple versions of spark installed just to match cloud :) , this should help. submitted by /u/jseeker528963 [link] [comments]

Beyond Chat History: Designing Portable Memory and Decision Boundaries for Enterprise AI Agents

Ticket #01028436 — No response after 10 days — Legal name correction request for data engenieer

How to Unit Test Spark Declarative Pipelines

When (if at all) does Databricks plan to have bidirectional sync between Lakebase and Lakehouse?

I’m working on a project that could really benefit from bidirectionally syncing table entries that come in through an application hitting Lakebase as well as those batched into the Lakehouse from an external API. Do we know when bidirectional sync is expected? submitted by /u/ceeej777 [link] [comments]

Genie Reading Jira Tickets

Has anyone figured out how to get genie to read Jira tickets? I point cursor/claude code at Jira and they fire away. However sometimes I’d like to develop with Genie. Is this possible? submitted by /u/Known-Delay7227 [link] [comments]

Load TBs into Postgres in Minutes

💡 NEW: Create Relationships in Databricks AI/BI Dashboard

Run jobs and pipelines as a group

You can now run pipelines as a group, avoiding dependency on a single principal (user or SPN). Databricks CLI 1.20.0 adds run_as.group_name at the bundle and target levels. The group needs its own workspace and data permissions; it does not inherit a member’s personal access. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Medallion Modelling Principles at Scale (Billions events, PB scale)

So, imagine a kafka ingestion into an initial bronze layer ingesting 2 billion events from a single topic. This data lands in a first bronze layer. The interesting part is that each event is deeply nested and semi structured. Each event once fully de-nested could contain 100s of data points. Typically you would land this data into bronze1 untouched; then perhaps another hop to bronze2 where the data stays 1:1 but typed etc My question is on the silver layer: do you explode nested parts of the event into entity tables (2B x 50entities for example) would be for example an entity table of 100B rows per day. This would be just one entity; possibly 10-20 entity tables in silver layer derived from this one payload/event Or....keep it 1:1 at silver also; keeping just the 2B rows in a table and ensuring nested types as Structs or Maps or Variant.....but losing the entity tables and possibly the query performance at read time Storage cost and query cost to be considered....as well as Gold layer preparation and ease of populating Gold layer... How to balance this at scale? submitted by /u/BigDaddyData187 [link] [comments]

Databricks Advanced Learning Festival: November 10 - December 08 2026

Genie Agents Added Support for MCP Servers

Databricks just added MCP servers to Genie Agents. This is super valuable and different than Genie One having access to MCP servers and let me explain why. Genie Agents are domain specific and targeted at certain business personas. Genie one is a general coworker that is anyone that is trying to improve their life with AI. So Genie Agents having the ability to answer questions about a certain domain AND having access to MCP servers like Microsoft 365, google suite, Atlassian,ect... allows domain level quetions to be answered with additional context besides structured data. It opens a whole new umbrella of questions that can be answered! I tried it out this morning and so far its just the managed MCP servers which are great but hoping they expand it to customer/external MCP. They worked exactly how expected though and better yet you can see the usage via Unity Gateway. Curious to see how others like this? Documentation linked here: https://docs.databricks.com/aws/en/genie-agents/external-tools submitted by /u/Wise_Ear_4064 [link] [comments]

CUSTOMER STORY | Headspace builds trusted AI analytics with Genie Agents on Databricks

I benchmarked Databricks against my iPhone

Not able to Log In to Databricks Partner Academy – Wrong Credentials After OTP

Is Enterprise Data Ready to Power AI?

What if Databricks Genie could do more than generate code?

HubSpot connector

Announcing: multiple triggers on Jobs (Beta)

Your Databricks job can now start from more than one condition. One job, several triggers (up to five of them for now). When any trigger fires the job runs. https://i.redd.it/6iiitfdej8uh1.gif Mix and match: Scheduled - on a timer File arrival - a new file lands Table update - a source table changes Model update - a UC model changes (Beta) Continuous - reruns as the last run ends Why it helps: stop cloning the same job for each trigger or building parent jobs with run-job tasks. Define the job and business logic once and attach every trigger it needs. Fix a bug once and every path gets it. Each trigger has its own pause, resume and delete. Set it up: open your job, go to Schedules & Triggers, click Add trigger, pick a type, Save. Repeat for the next one. Docs: https://docs.databricks.com/gcp/en/jobs/triggers#add-multiple-triggers-to-a-job What’s next: Higher trigger limits and activation windows (e.g. running a job continuously on weekdays while triggering it on table updates during weekends). submitted by /u/saad-the-engineer [link] [comments]

Genie Code chat history monitoring of other users

Genie keeps switching to serverless for code execution even though I have a classic compute for that

genie keeps falling back to serverless (which I don't have access to) and keeps failing code execution even though I keep switching it back to classic compute it's not helping any idea on fixes? submitted by /u/KannurLevelThreat [link] [comments]

My 2-Week Study Strategy & Resource Review: Shared in the Megathread

submitted by /u/Own-Green7779 [link] [comments]

Introducing session restore for serverless jobs

From Batch to Milliseconds: Real-Time Feature Views for Recommenders

Serverless pipeline with Materialized Views vs. Delta overwrite tables for ~60 Gold outputs

ABAC on Views is now in Beta in Azure Databricks!

Until now, ABAC row filters and column masks were primarily applied to tables. With this update, the same policy model can now be enforced when sensitive data is exposed through views as well. September 2026 - Azure Databricks | Microsoft Learn For example, a view such as sales.customer_view can expose customer data while a policy masks sensitive columns or filters rows depending on the querying principal. Here's a simple example of usage: CREATE POLICY mask_pii APPLY TO all_account_users EXCEPT admins ON tables.tagged sensitivity=high USING COLUMN MASK mask_email(email) Notice one thing - because Unity Catalog considers views as tables, use ON TABLE view_name to target a specific view. There is no separate FOR or ON VIEW syntax. I’m really glad to finally see this feature. I’ve personally been waitng for it for quite some time, since our team relies heavily on views as the consumption layer ;) submitted by /u/szymon_dybczak [link] [comments]

Thousands of tables in raw need to bring to Silver

I asked similar type of question before , i thought it will be 100's of tables now its 1000's, Very few high volume tables in few millions. but most of them are very small data. As per previous post reply suggestion i am doing yaml driven approach to load, thinking it will be max few 100. Now that they a re in 1000's it feels insane to configure so many tables in yml. Any alternative guys ? I already got feedback that sdp-meta sucks . How would you guys handle this , not all tables may need transformation but its good to have option to do transformation when needed when moving from raw to silver Edit: looking at the replies I wanted to mention that this is a on premises to cloud migration so each business unit will have there entire database migration to databricks . H once a large number of tables submitted by /u/Competitive-Fee-4006 [link] [comments]

Ai Parse Docs on 1 m docs

My org is ingesting a corpus of 1 million docs, estimated 9 million pages, of mixed complexity. Considering ai_parse_doc is a managed service, are we constrained by the speed of this function? Based on my research I assume this is true. Keeping that assumption, what would be the ideal compute specs to process the documents as quickly as possible? We’ve tried a few configurations (more memory, more workers) but have hit out of memory errors or MERGE errors and had to revert back to a config that is slower but reliable. We do have the gift of time right now, so not the biggest deal, but if we have to process under a tight timeline, I want to be prepared. I’m sure more information is needed but interested in folks that may have done a similar exercise and how long it took to parse a large corpus of docs? submitted by /u/dopedankfrfr [link] [comments]

"Organization permissions" in DAB's

Benchmarking Databricks against an iPhone 17 Pro

I did a fun benchmark, comparing various sizes of Databricks cluster against DuckDB on my iPhone 17 pro. A bit of a stunt but what it shows is just how fast modern CPUs are compared to the size of typical analytics workloads, and how efficient single-node compute engines are. https://preview.redd.it/07hjige5j4uh1.png?width=1344&format=png&auto=webp&s=e960b30f6e762c614b7c0368633587a6738e4e2d https://www.fivetran.com/blog/i-benchmarked-databricks-against-my-iphone submitted by /u/georgewfraser [link] [comments]

SharePoint Lists ingestion is now in Beta in Lakeflow Connect

Hey r/databricks , SharePoint Lists ingestion is now available in Beta with Lakeflow Connect. You can ingest structured data from Microsoft SharePoint Lists into Delta tables. The Beta supports two ingestion patterns: Ingest one SharePoint List into one Delta table. Ingest all Lists from a SharePoint site into one destination schema, with one Delta table per List. SharePoint Lists ingestion is available through the managed SharePoint connector API. You can create a pipeline using Declarative Automation Bundles or a Databricks notebook. UI-based authoring is not available in this Beta; we are working on adding it. To try the Beta, ask a workspace admin to open Previews , search for SharePoint Lists Connector , and turn it on. Tutorial Option 1: Declarative Automation Bundle This Declarative Automation Bundle configuration ingests one List: resources: pipelines: sharepoint_list_pipeline: name: sharepoint_list_pipeline catalog: main schema: ingest_destination_schema channel: PREVIEW ingestion_definition: connection_name: objects: - table: destination_catalog: main destination_schema: ingest_destination_schema destination_table: sharepoint_list connector_options: sharepoint_options: entity_type: LIST url: https:// .sharepoint.com/sites/ /Lists/ Option 2: Databricks notebook Step 1: Open the SharePoint example notebook , select Open in Databricks or Import Notebook , and import a copy into your workspace. Step 2: Do not modify cell one. Replace the pipeline configuration in cell three with the following, updating the placeholders: pipeline_spec = """ { "name": " ", "catalog": "main", "schema": "ingest_destination_schema", "ingestion_definition": { "connection_name": " ", "objects": [ { "table": { "destination_catalog": "main", "destination_schema": "ingest_destination_schema", "destination_table": "sharepoint_list", "connector_options": { "sharepoint_options": { "entity_type": "LIST", "url": "https:// .sharepoint.com/sites/ /Lists/ " } } } } ] }, "channel": "PREVIEW" } """ create_pipeline(pipeline_spec) Step 3: Click Run all to create the pipeline. Ingest multiple SharePoint Lists If there are multiple SharePoint Lists you want to ingest in the same pipeline, simply create a separate table object within the same pipeline spec definition and point the url at the Lists you're interested in. The SharePoint connector also supports ingesting all of the SharePoint Lists across an entire site. For site-level ingestion, follow the same steps above, but use a schema object instead of a table object and point the url to the SharePoint root site. Each List is written to its own destination table. Limitations: The current Beta does not support tenant-wide List ingestion nor SharePoint List attachments. See the documentation for complete Declarative Automation Bundle and Databricks notebook examples, including site-level ingestion. If you try it, let us know what kinds of Lists you are ingesting and where you run into friction. submitted by /u/BricksterJ [link] [comments]

Trouble while Maintaining Databricks Genie Agents (VS code extension available?)

Cant pay overdue bill and/or cancel subscription

Cant pay overdue bill and cancel subscription

I Benchmarked Databricks Against My iPhone

Validate new code before updating your production job with Databricks DABs

Test new code before updating your production job. With job_runs, immutable snapshots, and a validation dependency, a failed check can keep production on its previous code. See the full article: https://medium.com/databrickscommunity/databricks-dabs-ci-cd-4-ways-to-automate-your-deployments-6a732d9e928c https://www.sunnydata.ai/blog/databricks-asset-bundles-job-runs submitted by /u/hubert-dudek [link] [comments]

Move Databricks SCIM app between Okta tenants

We need to move our SCIM enabled Databricks instance from one Okta tenant to another. Asides from doing the basics below (push groups have been accounted for), is there anything I should be considering? Disable "Deactivate Users" in the current Okta SCIM app to stop deprovisioning calls to Databricks. Configure SCIM provisioning in the new Okta tenant against the same account-level SCIM endpoint. Enable provisioning in the new tenant; users will match existing Databricks accounts by email. submitted by /u/Dramatic_Point8556 [link] [comments]

Managed Omnigent on Databricks can't use external model providers (Az OpenAI,Gemini)in Unity Gateway

Can a Delta table have the “right” data but the wrong history?

Suppose the current data in a Delta table is completely correct. But when you check the Delta transaction history, you find unexpected UPDATE , DELETE , or MERGE operations, or records were changed and later restored. The final state looks correct, but the history of how the data got there is not. Would you consider the table reliable? What would you check first — Delta history, version history, operation metrics, or the upstream jobs that modified the table? submitted by /u/No_Ambition8323 [link] [comments]

Databricks SSH Tunnel: Connect Coding Agents and IDEs to your Workspace

Genie Code $150 in 5 days?

I have the following situation: I’m working on a client Databricks workspace, which has Genie Code enabled. However, as I am not an admin I cannot see how many DBUs I am using, in other words I cannot track/monitor my usage. The client has told me that they have set a budget limit per user of $150. According to Databricks Docs, price per DBU is $0.078. So this along with the free allowance of 150 DBUs, is roughly ~2000 DBUs So in my mind I’m thinking I got plenty to work with for the month. Suddenly I get the message “Limit reached, speak to admin…” after 6 days (from the beginning of the month). Is this possible? Did I burn through $150 in 5 days? I have been using genie code quite heavily, to help build a Databricks App, let’s say 3-4 hours usage on average for those 5 days with mixture of simple and complex tasks. As I don’t have a clue what my actual DBU usage is (due to not having permission, and it’s taking a while to get this information), I am having trouble believing if I do actually have a $150 limit or if this budget includes things like compute also. Or am I looking at this completely wrong and it’s entirely plausible I’ve used up the budget? As I try to get this information from them, I wanted to check if anyone else noticed anything similar as a heavy genie code user submitted by /u/Ibnsalaam [link] [comments]

How do you regression-test a Genie space or agent when a table or metric view changes underneath it?

Urgent Request: One-Time Exception for Databricks Certification Exam

rich colors affected by recent dark mode update

Zerobus Ingest with Kafka compliant API and WITHOUT a callback

Use Genie Code outside the workspace

Hey! I’m a Databricks product manager working on third-party development experiences. We just shipped Genie Code CLI , a new coding agent specialised for data and AI work. We’d love for you to try it out and share what you want to see next! Why we built this: Many of you prefer working locally with agents, but these agents might not always know the best way to work with Databricks. We wanted to provide an agent preconfigured to understand your Databricks data, pipelines, models, and apps. Use this when you want: Minimal setup and maintenance to use an agent that works effectively with Databricks. Governance provided through Unity Gateway. Access to agents outside of the Databricks workspace without model subscriptions or contracts. Getting started: (1) Install the latest version of the Databricks CLI . (2) Install Genie Code CLI : macOS, Linux, WSL: curl -fsSL https://github.com/databricks/genie-code-cli/releases/latest/download/install.sh | bash Windows powershell: powershell -ExecutionPolicy Bypass -c "irm https://github.com/databricks/genie-code-cli/releases/latest/download/install.ps1 | iex" (3) Start an interactive session by running genie in your project directory. Note that usage is billed through Unity Gateway’s model serving capabilities . For full details, check out the Genie Code CLI docs . Share your feedback: We’re releasing this fast to get your input asap. Please leave feedback and questions in the comments or reach out directly with a Reddit message! submitted by /u/BricksTrixTwix [link] [comments]

Change in DLT Pipeline Deletion Behavior associated Tables Not Being Deleted

You can now run an SDP pipeline as a group

A pipeline's run-as identity can now be an account group. Updates run with the group's permissions, and the tables the pipeline publishes are owned by the group. What it is Every pipeline runs as some identity. That identity decides what the pipeline can read and write, and who owns the tables it creates. Until now you could pick a user or a service principal. You can now pick an account group too. A Databricks role is a group under the hood, so setting run-as to a group also lets a pipeline run as a role. Why you'd want it If a pipeline runs as one person, it depends on that person's account staying active. When they leave and the account is deactivated, the pipeline can stop running and the tables it published can become inaccessible to everyone else. Service principals already fix part of this. Groups go a step further: a whole team owns the pipeline, and it keeps working as people join and leave the team. No owner reassignment when someone moves on. How to use it https://preview.redd.it/1ml56gev61uh1.png?width=1848&format=png&auto=webp&s=40275cf04ac3a837108289df8e8f16a855acfd98 In the UI: Open the pipeline, click Settings. Click the pencil next to Run as. Pick an account group you belong to, or one you have Assume permission on. Groups show up next to users and service principals. Click Save. The new identity applies on the next update. You can also assume a group’s identity to change / create a pipeline. And you can change the owner of a standalone MV/ST to a group as well! { "name": "sales-etl", "run_as": { "group_name": "sales-eng" } } Declarative Automation Bundles: resources: pipelines: sales_etl: name: sales-etl run_as: group_name: sales-eng What you need CAN_MANAGE on the pipeline. Either membership in the group, or Assume permission on it (for groups used for exclusive access). The group has to be an account group assigned to the workspace. Workspace-local groups don't work, and neither do the built-in admins, users and account users groups. Grant the group every permission the pipeline needs. Updates run as the group, so your own access to the source and data isn't enough. Docs: https://docs.databricks.com/aws/en/ldp/configure-pipeline#set-the-run-as-identity-to-a-group Please let us know what you think! submitted by /u/saad-the-engineer [link] [comments]

Databricks project structure: how do you organize metadata, sql, yml, etc

Hi everyone, I’m currently designing the structure of a Databricks project using Declarative Automation Bundles and I’m interested in how others approach the organization of their codebase For a generic example, imagine a data platform with several domains, such as Sales, Customers, and Operations. Each domain has Bronze, Silver and Gold tables. I’m trying to answer a few questions: 1. How do you organize your Bundles / YAML files? For example, would you structure it something like: sales/ databricks.yml resources/ bronze/ silver/ gold/ customers/ operations/ Or would you have a single databricks.yml and organize the domains underneath it? And within resources/, would you have one YAML per job, per pipeline, per domain, etc.? 2. How do you define table and column metadata? For example, suppose I have a SQL transformation that creates a Gold table. Would you define the comments directly in the transformation SQL, for example using column COMMENT definitions? Or would you maintain metadata separately, for example: metadata/ customer.sql sales.sql containing COMMENT ON TABLE and ALTER TABLE … ALTER COLUMN … COMMENT statements? I like the idea of separating transformation logic from data documentation, but I’m wondering what people actually use in production. submitted by /u/Quantumizera [link] [comments]

Optimize Delta Lake Write Performance With spark.databricks.delta.optimizeWrite.enabled

Hey, I work as a Databricks Engineer, so here's my take. Small file problems destroy read performance in Delta tables. If your streaming pipelines or frequent batch inserts generate thousands of tiny parquet files, enable spark.databricks.delta.optimizeWrite.enabled at the session or table level. When set to true, Databricks automatically coalesces data right before writing, targeting roughly 512MB per file. This happens dynamically within the shuffle stage, reducing the need for post-hoc OPTIMIZE commands. Pair this with spark.databricks.delta.autoCompact.enabled to automatically compact small files in partitions after writes complete. You will trade a fraction of driver memory and shuffle time for significantly cleaner storage layouts, faster merge operations, and much lower cloud storage list-request costs during downstream reads. For anything else on Databricks | Platform Engineering | AI , visit Abilytics — AI Systems, Data Engineering & Platform Engineering Services. submitted by /u/AbilyticsEng [link] [comments]

Avoid DLT infinite loops by targeting specific table names

Hi, Databricks Engineer at Abilytics here. If you use Delta Live Tables with wildcard streaming reads like spark.readStream.table("events_*"), your pipeline can trigger unexpected full refreshes or infinite dependency loops when DLT infers schema evolution on newly matched upstream tables. Instead, explicitly declare your dependencies using the live schema namespace. Reference tables directly by their exact identifier, such as spark.readStream.table("live.events_bronze"). This ensures the DAG resolver builds a deterministic execution graph without misinterpreting wildcard patterns as dynamic schema changes during optimization. If this is useful, there's more on Databricks | Platform Engineering | AI at Abilytics — AI Systems, Data Engineering & Platform Engineering Services. submitted by /u/AbilyticsEng [link] [comments]

Mastering SQL for Data Analytics: A Step-by-Step Roadmap

Supervisor Agent Visualisations

Time-Travel on Tables with ABAC RLS/CM - Beta

Hey all! I’m a Product Manager at Databricks who just launched Beta support for time travel on tables with ABAC policies. You can now query historical data using VERSION AS OF or TIMESTAMP AS OF while keeping ABAC row-filters and column-masks enforced. The main design question we faced was how policies should interact with historical schemas. We chose a security-first model: Load the requested historical snapshot. Match it against the current schema. Validate the match. Renamed or dropped historical columns block the query to prevent data leak, while columns added later are excluded from the result. Apply the active ABAC policies to the historical data and return query results. Here's an example: How time-travel works on ABAC-protected tables This fail-closed behavior is especially important when a policy references a column that did not exist historically. Treating the missing column as null or assigning a default could change the row filter’s result and expose restricted rows. Databricks blocks the query instead. Why fail-closed behavior is critical I'm curious if the community has any feedback on the approach. Any input will be valuable as we step towards the GA milestone. For more information, please see our documentation . submitted by /u/alicks22 [link] [comments]

Configure Unity Gateway rate limits for system.ai model services

I am trying to configure rate limits for system-provided model services under system.ai where I am unable to set the rate limit for the model. Below is the scenario I tried When I open Unity Gateway, select system.ai.claude-sonnet-5, and try to configure the rate limits, the UI displays: "User does not have permission to access this resource." I see the same behavior for other system.ai models. I have Account Admin access at the Databricks account level. However, when I run the following SQL command: SHOW GRANTS ON SCHEMA system.ai; the result only shows the following permissions for account users: EXECUTE READ VOLUME SELECT USE SCHEMA There is no MANAGE privilege shown on system.ai. Additional observation When I directly call system.ai.claude-sonnet-5, I receive HTTP 403 with the following message: PERMISSION_DENIED: The endpoint is temporarily disabled due to a Databricks-set rate limit of 0. I am not sure whether this is related to the inability to configure the rate limit in the UI or whether it is a separate platform-side restriction. Could someone please help me to understand the distinction between account-level administration and Unity Catalog / metastore-level permissions for system.ai model services, and understand the correct least-privilege configuration needed to manage Unity Gateway rate limits?

Is the Data Engineer Learning Plan enough to pass the Databricks Certified Data Engineer Associate e

Best practices for DR & Checkpoint Management of Serverless Declarative Pipelines (ST/MV)?

Sybase connectors

Are there any Lakeflow connect connectors for SybaseIQ? submitted by /u/RazzmatazzLiving1323 [link] [comments]

Practical Generative AI Skills for Beginners and Working Professionals

Request for 50% Certification Voucher Discount

Stop Hiding Your Credentials in Notebooks: A Look at Secrets in Unity Catalog

Run a job on every Databricks DABs deployment with job_runs

Databricks DABs can run a job every time you deploy. Use job_runs with on_bundle_deploy: true for deployment checks, logging, or configuration updates. See full article: https://www.sunnydata.ai/blog/databricks-asset-bundles-job-runs SunnyData https://databrickster.medium.com/databricks-dabs-ci-cd-4-ways-to-automate-your-deployments-6a732d9e928c submitted by /u/hubert-dudek [link] [comments]

Announcement: Environment variable support for serverless notebooks and jobs

Hey all, you asked and we listened - env var support is in beta for serverless notebooks and jobs! Have you been wanting to run DBT tasks but need environment variable support? This feature is for you! ✅ Do your whl or notebook tasks just need a way to inject environment variables at runtime? This feature is for you, too! ✅ Do you just love the environment? That's awesome, me too, but this feature won't help you with that. ❤️ 🌎 How to try? Ensure that the feature is enabled in the preview portal (with admin perms if it isn't already)- grep on ` Environment variables in Lakeflow Jobs` How does it work? Jobs UI > Click `Select environment variables` > `+ Add environment variables`. https://preview.redd.it/vf3y80jbvwth1.png?width=1636&format=png&auto=webp&s=e68cec5c30394ecb5b982f3809c88fc387414a1f Then create a name `production` for the env var collection (that way you can reuse it on multiple tasks), fill out your key-value pairs, and/or or paste in the file path to a .env file. Note: We will add a file picker for choosing the .env file in an upcoming release - thanks for being patient https://preview.redd.it/8c606502vwth1.png?width=1762&format=png&auto=webp&s=5ac9bca037c793c8758ce8b4625fb26b28a16a1a Jobs API > Define a collection via a key e.g.,`production` and use it on many tasks. "tasks": [ { "task_key": "process_orders", "notebook_task": { "notebook_path": "/Workspace/Shared/process-orders" }, "environment_key": "default", "environment_variables_key": "production" } ], "environments": [ { "environment_key": "default", "spec": { "environment_version": "5", "dependencies": [] } } ], "environment_variables": [ { "environment_variables_key": "production", "spec": { "variables": { "APP_ENV": "production", "REGION": "us-west-2" }, "files": ["/Volumes/company/default/prod/execution.env" , "/Volumes/company/default/prod/format.env"] } } ] ] Finally, consume it in your code: 🤓 import os app_environment = os.environ["APP_ENV"] Temporary limitations : these will be fixed within a week or so - I'll update this post Env vars will not propagate to regular UDFs (aka non-UC) Defining an env var's value from a secret Learn more : https://docs.databricks.com/aws/en/jobs/environment-variables Provide any and all feedback. Also, what other features you'd like to see? (Asking for a friend as roadmap planning is starting soon!) Thanks! submitted by /u/justinAtDatabricks [link] [comments]

DevOps Fundamentals: General problem with SCORM modules being marked as completed

DevOps Fundamentals: Lecture - Introduction to Unit Tests for PySpark

DevOps Fundamentals: Lecture - Planning the Project

Unlocking High-Frequency Workloads with Nanosecond-Precision Timestamps

Ubunye Engine – One Pipeline, Local/Databricks/AWS/Azure/GCP

Practical Enterprise Blueprint for Greenfield impementation

What is the closest thing to a practical, opinionated enterprise Azure Databricks blueprint? I’m not looking for docs on individual features. I know Databricks, IaC, UC, Asset Bundles, CI/CD etc. very well (seasoned architect, DBX engineer professional). What I’m missing is how experienced Databricks architects/FDEs usually put the whole thing together for a greenfield setup. Things like: repo split: Azure/IaC, UC, platform, data products (how many repos, how to exchange secrets, how may MIs) dev/test/prod topology (catalog and workspace relation) managed identities and ownership model UC structure, RBAC vs ABAC (federated IAM setup) what belongs in one Asset Bundle, how many bundles, how to deploy/orchestrate them (monorepo or not, asset bunle per source acqusistion/data product or broader) CI/CD and environment promotion (branching strategy, deploying multiple platform team vs data product responsibilities ingestion/integration/orchestration patterns The official docs are good, but mostly describe these things in isolation. Is there a solid repo, reference implementation, blog series, conference talk, consultancy blueprint, or even a good Reddit/forum thread that shows the whole platform design ? Basically: if a Databricks FDE started a vanilla enterprise Azure setup tomorrow, what would the default starting point look like? Mostly interested in high friciton points where reality of operating platform meets "best practice". My focus is "classic" data warehousing, rather ML/AI/apps. submitted by /u/wicked_medusa [link] [comments]

Auto stop option on databricks apps

Want to request a feature for databricks app, it should have an auto stop option. I know Databricks markets it as a web app that should be readily available, but I think customers deserve the option to turn auto-stop on or off, like we do for, say, SQL Warehouses. This has been a major blocker for us suggesting apps to leadership, the engineering team has build some really useful apps for usecases like Data modelling and stuff which is used when needed but they running 24*7 all time is wasting compute and getting flagged in finops P.S i know we can have a databricks job which can shut down and start the app during working hours but we have a global team working round the clock so it's not feasible for us. If we have someone from databricks apps team this is a highly requested thing from alot of people around the community. submitted by /u/sumit671 [link] [comments]

What changed on Lakenaut after your feedback, and the site is now an MCP server you can plug into Claude

First of all, I really want to thank you everyone commented my latest post and encourage me to keep update and mantain the website. For the ones who miss my post is this one (for reference): https://www.reddit.com/r/databricks/s/S06gTMmm2P A few of you wrote in over the last two weeks, and most of it is live now. The short version, then one bigger thing. From the feedback "Too many categories, the docs' hierarchy is clearer." The sidebar groups now follow the Databricks documentation's domains (Platform & governance, Data engineering · Lakeflow, Data warehousing & BI, AI & ML), while each area keeps the name the workspace menu gives it. The sidebar is contextual: the doors are always there, the tree opens only where you are. "Finishing a page and moving on is one gesture." It is: Mark as done and Done, next sit together at the foot of every concept. "I want to know what moved." Every concept, area, path and roadmap now opens with a "What changed, and when" box, and /changes/ is the whole log by day: which page, what was done, and whether a person did it by hand or accepted a correction the agent drafted after the documentation changed. Every end-of-support row has its own address now, and search lands on it instead of on the list. A fifth way through the material is taking shape: practices, decision pages filed on the well-architected pillars. When to pick a thing, for whom, what choosing wrong costs. The first one is in draft. The bigger thing: an MCP server, no key Put the MCP URL (page below) into Claude, ChatGPT, Cursor or Claude Code and the assistant reads the published site instead of guessing: 170+ concepts, 8 paths, 5 roadmaps, the news, the resources, and the platform tables (what is in preview, what was renamed, what is ending). Thirteen read-only tools, 60 calls a minute per address, every answer links back to the page it came from. How to connect it: https://lakenaut.dev/developers/ Two questions If you plug it into an assistant, what is the first thing you ask it? I want to know whether people reach for prep, for "what is in preview", or for something I have not thought of. Is there a kind of page you keep wishing existed? The practices came from one comment. The next kind could come from yours. Again, thank you and let's connect! submitted by /u/imthenitto [link] [comments]

hii this my fist post to the community

Account Admin unable to configure Unity Gateway rate limits for system.ai model services

Genie Agent can't be added to git repo or DAB. Nothing working.

Has anyone tried working on a Geni Agent, and then tried committing those results into git? Admittedly I'm a bit new to Databricks, and to DABs, but this story is missing, and the whole experience is quite unintuitive . Any developer generally expects to be able to commit their work to source control when they've invested their time. But I'm finding that incredibly hard, when it comes to these Genie Agents. Obviously the first thing I tried to do is clone the agent into a git-controlled folder and push it to git. That didn't work. It says the asset type is unsupported. submitted by /u/SmallAd3697 [link] [comments]

How to find the reason for a slow databricks job?

I want to know how ppl usually find the reason for a slow databricks job. For example, if a spark job normally takes 20 mins but suddenly takes an hour, what do you check first? Do you look at the joins, partitions, shuffles, amount of data, cluster settings, or the spark UI? How do you figure out whether the problem is with the code, data, or the cluster? Are there any common issues you guys have seen, like data skew or too many small files, that can make the job much slower? I want to know what the usual approach is when a Databricks job suddenly becomes slow. submitted by /u/Klutzy_Solid5200 [link] [comments]

Free, Hands-On Databricks Training: APAC Learning Festival

Databricks Community Champion - September 2026 - Venugopal Dabbara

What Is – Agent mode in Genie? (Quick answers vs investigations)

Does anyone has databricks 100% voucher code?

Data Classification on Foreign Catalog

Solution Accelerator Series | IOC Matching and Multicloud Query Federation

Need assistance escalating Databricks Academy account transfer (Ticket marked closed / unmonitored)

Export PDF option not available in iframe with scoped token databricks/aibi-client

Why Databricks can't match ClickHouse Cloud for real-time analytics

Recommendations for limiting spend on a pay-as-you-go account

Do you have any advice to limit spend in Databricks as a personal user? I'm trying to understand gotchas on a pay-as-you-go account. It's to try out the generative ai features in particular (knowledge assistant, ai endpoint, agent bricks, among others). The ideal would be "no more than X concurrent DBU consumption" policy across the board, but I'm not seeing that. I'm coming up against limits of Databricks free edition for those features, so was wanting to set up a pay-as-you-go account, but while reading, I'm seeing more and more ways to spend money in the platform. On Fabric it was a single F2 capacity I could start and stop. Databricks has serverless in like 5 spots so far that I've seen, and I'm trying to understand what my spend risk is before I sign up. I had a trial a year ago that's expired. Also knowledge assistant isn't available on the trial. My worry is that while doing the study guides, it ends up that I have 10 different 4-6 DBU serverless endpoints that run for 2 days before the billing catches up. $20 is fine, $200 is annoying, $2000 is unacceptable. submitted by /u/NiceTryAmanda [link] [comments]

Serverless DNS requests to external API being DROPPED despite FULL_ACCESS network policy and no bloc

Course language change request: Databricks GenAI (Korean to English)

Announcement | Lakebase Search: State-of-the-art full-text and vector search for Postgres

Rebuilding an 800-Line PySpark Pipeline in 200 Lines of Databricks SQL

How SQL and Python Enhance Data Science Learning

Group-based auth with Lakebase Data API

What is the most effective way to evaluate Genie Agents/ Genie Spaces.

Turning P&C Claim Documents into Structured Data with Databricks AI Functions

Did Databricks just enable Automatic Identity Management in Azure?

We have used SCIM for user provisioning for years, but after 1st October 2026 the Automatic Idenity Management was suddenly enabled in our account, and lots of conflicts with SCIM provisioning is logged I cant find any announcements on this submitted by /u/No-Leather6291 [link] [comments]

The part of data pipelines we are thinking more about

submitted by /u/BullfrogFinal3566 [link] [comments]

What makes a Databricks data platform truly AI-ready?

Delta Table or Materialized View? Here’s how I actually decide

Genie Code vs. Omnigent: How to structure workspace instructions, AGENTS.md, and skills without breaking production

Following up on recent discussions around Genie performance I've come across... a lot of us have a similar frustration with Genie. Context bloat, prompt drift, and agents making destructive errors (Junior mistakes like setting a mode to overwrite). Part of this is limitations of genie, some is instruction "stuffing" and some is using the right tool for the right job. Also, guidance on how to structure and create instructions have changed a lot over the summer. With Omnigent being available alongside Genie Code, here is my opinion on what should separate these tools and how to structure context files to keep agent outputs deterministic. I made a table because post sprawl is real... Genie Code vs. Omnigent These are not competing options. They solve problems in different places and scales: Layer Genie Code Omnigent (Meta-Harness) Primary Role Native in-workspace assistant for notebooks, SQL Editor, and Lakeflow Pipelines. Unified meta-harness to run, swap, and persist multi-agent workflows (Claude Code, Codex, Cursor, PI, Hermes) across local runners and cloud sandboxes. (Also, try out Debbie that has the ability to use a /debate skil) Governance Prompt-level instructions, workspace instruction files, and Unity Catalog user permissions. External, stateful policies written in CEL (hard cost limits, network sandboxing, pre-commit git checks). Best For Interactive notebook debugging, quick PySpark logic, end-to-end Lakeflow pipeline scaffolding. (Think StackOverflow, for those of us of that still remember it) Multi-repo refactoring, team co-driving sessions (eg. "Multi-player" agent sessions), and running autonomous CLI agents with guardrails. Execution Model Synchronous, interactive execution tied to your active workspace session and compute. Asynchronous, persistent execution that runs in the background (survives laptop close) with shareable web/desktop sessions. My Rule of thumb: If you are working interactively / ad-hoc in Databricks and need help with syntax, where something is, why something is failing, use Genie Code. If you are doing anything complex, want to be able to get one harness to check another one (like Claude Code checking Codex's response/PR) and need programmatic guardrails that can't be bypassed by prompt injection, go use Omnigent. And if you are using Genie and want it to read your "instructions": Databricks does not natively parse an arbitrary instructions.md dropped in a folder. If you want repo/project-level discovery, the standard Genie Code looks for is AGENTS.md (or CLAUDE.md ). Shoving context for "multiple project islands" into one file guarantees prompt dilution. Workspace defaults ( .assistant_workspace_instructions.md ) belong in under 30 lines. Scoped project rules belong in AGENTS.md . Task-specific behaviors belong in modular skills under .assistant/skills/ called via @skill-name . Andrej Karpathy created a git repo "karpathy-guidelines" that walk through some of the overarching concepts he recommends around creating skills and instructions. Specifically one of the ones that may come as a surprise is that if you repeat a instruction point it will lower the effectiveness of the instructions and cause degradation. submitted by /u/OkImprovement7010 [link] [comments]

Solutions Architect Final Round (Build Demo Pitch Round)

Hi all, I have my final round for a Solutions Architect role (the build demo pitch round) coming up this week, and I'm trying to understand what to expect. Are the prompts mainly focused on proposing a data ingestion pipeline / medallion architecture or is the task more complex? Is the focus of the round the code and architecture itself or presentation / delivery skills? Any help would be much appreciated. submitted by /u/mesathink [link] [comments]

Guidance Required: Scheduling a Biweekly Databricks Job