コミュニティが今、聞いていること

Error with Databricks ODBC Driver OAuth 2.0

CUSTOMER STORY | Powering Up: EnergyAustralia’s race to real-time

Databricks Serverless Compute: Practical Recommendations, Cost-Performance Guidance, and When to Use Serverless Instead of SQL Serverless Warehouses or Classic Clusters

Databricks Serverless compute is a snappy choice for every-day queries with some cost trade-offs. For ETL workloads, it is both fast + cost-efficient. In either scenario and unlike common misconception, there is no cluster fine-tuning needed. Some more detailed takeaways from some testing on my coffee dataset + practical guidance below. Quick disclaimer: I work for a Databricks aligned consulting company. All reporting and recommendations here in any direction are independently made. General Observations from Performance Testing 1) Speed wise, much to my surprise, Serverless beat some of my previous Databricks’ SQL Serverless performance testing. Not across all queries, but in nearly half. 2) Databricks’ SQL Serverless compute is still the best compute option from a pure cost-perspective when it comes to “every day” analytics needs, but from an ETL perspective, “Serverless” is very comparable, without the trial & error of even doing t-shirt sizing, let alone knobs. 3) The scaling time to support an initial large workload was noticeable on the first query and does have room for improvement. I did a dummy print() statement right before (which took like 1-2 secs to warm up). Then hitting that first query containing the joins and 7B+ rows fact table clearly gave the engine something to chew on. For reference, on my SQL Serverless test a while back, the same query took about 20 secs on a large, 15 on XL. Practical Serverless Recommendations 1) If I was orchestrating ETL workloads with SQL and/or Python, I’d use Serverless compute as the cost of Serverless for jobs is pretty darn affordable. My concerns around costs are completely negated when it comes to jobs compute. For new ETL workloads, Serverless is an easy choice for me. For existing ETL workloads, I’d try out to see if Serverless lowers your costs and/or improves performance. Also explore any potential compatibility issues. If you have heavily fine-tuned workloads, that don’t really require much maintenance, migrating the workload might not be best. 2) For daily Python work on established Databricks environments, I would consider trying Serverless. Not having to wait for clusters to boot up and fine-tuning is likely going to lead to productivity gains offsetting any higher costs from interactive Serverless compute itself. No more “left the cluster on by accident” or “I oversized it” type of situations. That said, there are orgs with very experienced folks where cluster management feels like second nature to them. 3) If I was exclusively working with analytical SQL workloads, I’d still use Databricks’ SQL Serverless compute for my day to day queries + dashboards. 4) For folks new to Databricks, I would generally stick to “Serverless” compute all around, and use the “SQL Serverless” warehouses only for BI type experiences. Closing Thoughts on Serverless Having very performant compute without worrying about any configurations (other than rate limits to give you cost controls if you wish) is quite nice from a productivity standpoint. No node count, DBR versioning, t-shirt sizing, nothing. You have plenty of compute options, with or without fine-tuning ! I stress this "no fine-tuning" because there is a plethora of LLM generated articles outdated on Databricks' compute options. Thank you for reading! *No AI was used to write or assist in writing this post. For better or for worse. **Dataset and queries used, configured for the 7B+ row count: https://github.com/joshbogran/coffeeshopdatageneratorv2 submitted by /u/JosueBogran [link] [comments]

RoadMap DE

How to think about interactive vs jobs clusters?

My understanding is that jobs clusters are cheaper (by ~50% or so?). They are a bit more challenging to use (from unattended code). I would like to use them for "spark connect" but am anxious to try it given that I've been told the jobs clusters won't be compatible. I was told that they might work, but the cluster might then die prematurely before remote clients are finished with them. I am under the impression that "interactive" clusters are for analysts and data scientists who are doing collab work together in notebooks. My spark connect workloads would NOT be comparable to that scenario. The workloads are more analogous to unattended jobs, that are simply triggered remotely. I'm not optimistic about using jobs clusters for this type of scenario ("spark connect"). I may be forced into using the "interactive" clusters for spark connect. It is unfortunate that the useful scenarios for jobs clusters are so limited. How are customers supposed to feel about databrick's strategy for these "jobs clusters"? Should we think of them as an inferior type of cluster that won't work properly for the interactive workloads AND for many types of unattended workloads? Is Databricks explicitly trying to make them unfriendly to use, so that customers are forced to use the more expensive "interactive" compute clusters? I think the API for jobs clusters has been slightly improved in the past few years (eg. to add multiple tasks on a single job). But the progress has been slow and I don't have much hope for using "spark connect". I'm fairly new to databricks and would love to be proven wrong about this. submitted by /u/SmallAd3697 [link] [comments]

Databricks Jobs to Kubernetes Spark Jobs migration

I'm discussing some architectural changes with my data team, and one of them is about cutting costs. The idea is to migrate Databricks Jobs to Spark jobs on Kubernetes. We've done a PoC and it works well. But despite the clear cost savings, I'm getting pushback from management and they're resisting the change. Can someone explain the reason behind this? It seems clear that you pay double for running jobs in Databricks, and you could save a lot of money. Yes, doing this means we bypass Unity Catalog, but we could convert only the Bronze and Silver layers. submitted by /u/worldpwn [link] [comments]

At what data size do you stop reaching for Spark?

Photon enabled but a large share of the plan is falling back, cost up and runtime flat

Beyond the Dashboard: How Transferz Built a Truly Data-Driven Company Culture With AI/BI Genie

Streaming read fails with "Detected a data update" after a restatement job touches the source

Preventing Duplicate Records When Reprocessing Data in Databricks

Secrets Redaction seems to work only when the retrieved value is assigned to a varible

5 Ways to Enhance Databricks AI/BI Dashboard Tables with HTML

Choosing between a materialized view and a job-maintained Delta table for an expensive SQL query

What do you think about investing in Databricks at its current valuation?

submitted by /u/Dry_Document_9136 [link] [comments]

Notebook commands hang after restarting a cluster with custom library installed

Workshop, Sep 19: build explainable AI apps with Neo4j, GraphRAG, Cypher and LLM Agents

There's a hands-on workshop on September 19 relevant if your team runs its data platform on Databricks and is looking at adding an explainable, graph-backed RAG layer on top for reasoning over structured and unstructured data. You build: A knowledge graph in Neo4j that becomes your single source of truth for an agentic RAG system Agentic retrieval combining vector search, keyword search, and graph navigation Multi-step, verified entity and relationship extraction Text-to-Cypher for natural language graph querying Explainable, source-traceable answers with evaluation built in Runs on real financial filings and news data, working through a complete production use case. Led by Dr. Alessandro Negro, Chief Scientist at GraphAware, bestselling author. Link if you want to check it out submitted by /u/camerongreen95 [link] [comments]

ODBC setup

Databricks architecture drawn step by step: control plane, compute plane, Delta, Unity Catalog

Announcement | A practical approach to end-to-end Solvency II reporting in Databricks

Is the data catalog finally dead?

RIP data catalogs! Source: https://contextandchaos.substack.com/p/is-the-data-catalog-finally-dead For twenty years, data catalogs have been judged by coverage. How many assets had a description, an owner, a lineage graph. Coverage is easy to count. It never told anyone whether any of it actually worked. That was fine when a person was the one reading it. If someone hit a bad definition, they shrugged, asked a colleague, and moved on. Nobody wrote it down. So the catalog never heard about its own mistakes, and never got better. Agents don't shrug. They take that definition, run with it, and hand back an answer that's confident and wrong. Thousands of times a day. Which raises a different question. When an agent gets something wrong, what actually failed, the context, the data underneath it, or the way it got retrieved? That's a completely different thing to measure. And honestly, most teams can't yet. submitted by /u/growth_man [link] [comments]

Announcement | Unifying governance across engines and catalogs in the Open Lakehouse

Lakeflow connect CT pipeline keeps running multiple connections in source

A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks

ConcurrentAppendException on parallel MERGE into the same Delta table, despite disjoint keys

Solution Accelerator Series | Customer Entity Resolution

AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns

Could a “Data → Agent” composer be useful for Databricks?

I've been thinking about a gap between Databricks data and agent frameworks. Databricks already has a lot of the building blocks: - Unity Catalog - Genie / Genie Agents - MCP - Vector Search - AI Gateway - Agent skills/tools - MLflow - Omnigent / Kasal And tools like Omnigent and Kasal already solve a lot of the agent orchestration/execution side. But I'm wondering about the step before that: What if a customer already has a large, curated and governed Databricks data estate — how do we turn that data estate into an agent-ready configuration without manually wiring everything together? Something like: Existing Databricks Data Estate ↓ Data-to-Agent Composer ↓ ┌────────┼─────────┐ ↓ ↓ ↓ Domains Semantics Metrics ↓ ↓ ↓ Genie MCP Skills └────────┼─────────┘ ↓ Agent Configuration ↓ Omnigent / Kasal ↓ Agent The idea wouldn't be to build another chatbot or another agent framework. It would be a Databricks-native composition/bootstrapping layer that understands an existing Unity Catalog/data estate and generates the pieces needed for agents to work with that data — domain boundaries, semantic context, approved tools, Genie configuration, MCP exposure, skills, policies, evaluation setup, etc. In other words: Kasal/Omnigent: Agent → Tools/Data Proposed layer: Data Estate → Agent I'm curious if this is already solved somewhere in the Databricks ecosystem, or if people are currently doing this manually when building enterprise data agents. Would love to hear how others are approaching the “existing data estate → production-ready data agent” problem. submitted by /u/imsuryya [link] [comments]

[Discussion] Lakeflow Jobs: If your data catalog could trigger a job on any change what would it be?

In addition to triggering on time (cron) intervals you can trigger on data landing: files arriving on a share, table gets a new commit and (soon) jobs completing upstream. While we were building Model update triggers (which trigger when UC registered models change), we were discussing the possibility of triggering on any entity in UC’s metadata changing. Examples here could be: Someone tags a column PII and you want to trigger a scan or retention job Someone changes a grant and an access review job gets triggered Someone is given SELECT permission on a table More generally it could be: Schema change on an upstream table (column added, dropped, renamed, type changed): run compatibility tests, rebuild the downstream model, ping the owner before it breaks. PII or sensitivity tag added: kick off a scan, apply retention, open an access review. Grant or ownership change: access recertification, sync to the entitlement system, log for audit. Table created, renamed or dropped in a schema: auto-register or clean up downstream assets. Questions: Which of these would you use and what are just noise? What kind of change would you really really want that you currently do by hand or after the fact? Anyone doing this off audit logs, system tables or just polling running jobs? submitted by /u/saad-the-engineer [link] [comments]

Name mismatch confusion

Databricks voucher

Failed to resolve External API

Feature enablement for Foundation Model Unity Catalog permissions

Built a multi-jurisdiction tax engine & compliance ledger on Cloudflare D1. Benchmarking 0.2ms P50 latency.

Wanted to share real-world telemetry from a lightweight micro-ledger I deployed to replace bloated ERP data entry for cross-border trade (GCC / East Asia). The Stack: • Compute: Cloudflare Worker (handling input validation, API auth, and edge UI hydration). • Persistence: Cloudflare D1 (SQLite at the edge). • Payloads: Multi-jurisdiction tax adapters (ZATCA 15% SAR, FTA 5% AED, CN VAT & Export rebates) logging immutable audit trails. The Numbers from Edge Telemetry: • P50 query latency: 0.2ms – 0.3ms • P99 query latency: 2.1ms • Idempotency: Implemented key checks to eliminate duplicate submissions from unstable mobile networks. Running an immutable, auditable transactional ledger directly at the edge with zero cold starts and $0 infrastructure overhead. Anyone else running high-throughput relational ledgers directly on D1 instead of offloading to external Postgres/PlanetScale? submitted by /u/DevIntheDark-33 [link] [comments]

Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed

Built a Databricks medallion pipeline for NYC Taxi data

Been working on this as a way to get hands-on with Databricks Asset Bundles and Unity Catalog governance. It's a migration of an old on-prem NYC Taxi analytics stack (ClickHouse + Spark + Docker + Terraform) into a proper Bronze/Silver/Gold lakehouse. A few things I focused on: Auto Loader for incremental ingestion, triggered by file arrival Data quality handling that doesn't just drop bad rows — duplicates, zero-distance trips, and reversed fares go into dedicated quarantine tables instead of being silently discarded Databricks Asset Bundles for deploy/orchestration (dev + prod targets) 3 published AI/BI dashboards on top of the Gold layer (fleet ops, finance, compliance) It's intentionally small in scope — meant to demonstrate the lakehouse pattern, not be a production-scale platform. Currently only Green Taxi data; FHV comparison is planned next. Repo: https://github.com/Hamza-Bouali/NYC-DATABRICKS Would love feedback, especially on the Silver-layer data quality rules or the bundle structure open to critique. https://preview.redd.it/ej6hvzrbokph1.png?width=1667&format=png&auto=webp&s=b090b2c5165efcc83fb3f2b5c38ff6ef34c8ef48 https://preview.redd.it/rih9a5sbokph1.png?width=1879&format=png&auto=webp&s=1b85b90608e1b1273da4f8cbb4fa65d9f2c397a7 https://preview.redd.it/kjdor4sbokph1.png?width=1687&format=png&auto=webp&s=677c6194ede0cd01a8630ee98f85d11a5146b410 submitted by /u/No-Pollution-2274 [link] [comments]

SAP Medallion Architecture Patterns: What Bronze, Silver and Gold Have to Do Differently

Why Unity Catalog Managed Tables are recommended

Unity Catalog managed tables are the best choice you can make but do you know why? In the second episode of SuperSkills Oleksandra Bovkun and I demystify all the reasons to help you make your choice. Link to the video: https://youtu.be/Q7y8\_bSfVjQ submitted by /u/Youssef_Mrini [link] [comments]

Slow Running SQL Query

CUSTOMER STORY | Moving beyond with Data + AI-driven service intelligence at TK Elevator

Databricks Community Champion - August 2026 - Shamen Paris

Community BrickTalk | Real-Time Data & AI: Tripwise Demo

Hey r/Databricks ! Join us for community BrickTalk on Thursday, September 24 , focusing on real-time data streaming, AI agents, and governance using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Data and AI, giving customers a direct line to the people behind the products. In this session, we'll walk through a live demonstration of the Tripwise Demo , featuring: Sub-Second Transactions & Streaming: Device registration into Lakebase with sub-second reads/writes, plus telemetry streaming via Zerobus through a governed Medallion architecture. AI-Generated Offers & Pricing: Generating real-time agent offers using Foundation Model APIs and scoring behavioral data for usage-based renewal pricing. Natural Language Analytics: Enabling underwriters, product managers, and marketing teams to query governed insurance data in seconds using AI/BI Dashboards and Genie. Unified Governance: Managing safety, compliance, and control end-to-end with Unity Catalog. This is a great chance to see real-world architecture in action and ask questions directly to Databricks experts. When: Thursday, September 24 9:00 AM PT 12:00 PM ET 5:00 PM BST (London) 9:30 PM IST Register here and save your spot submitted by /u/Subject_Ant1789 [link] [comments]

data contracts

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on. Within a team, its easy enough to define a contract. We codify it and always check that the data matches. But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing? I want to keep this stupidly simple and enforce it automatically. How do you do this in practice? submitted by /u/ptab0211 [link] [comments]

Found 847,293 files in a production Delta table. Broke down what it was actually costing us.

Streaming pipeline had been running fine for months. Every micro-batch landed clean. Then a GROUP BY with a date filter started timing out, and DESCRIBE DETAIL showed numFiles at 847,293. That's a 600 GB table averaging 750 KB per file. Recommended range is 128 MB to 1 GB, so we were sitting at something like 0.6% of the lower bound. "Small files are inefficient" undersells it. The actual problem is driver-side file listing. Spark keeps a file listing in memory for every query plan, and with 847,000 files the driver has to process all of it, single-threaded, before it reads a single byte. Doesn't matter if you throw a 64-core cluster at it. Three minutes of listing before any parallelism even starts. I actually sat down and priced this out (back of envelope, didn't model autoscaling properly, but the order of magnitude should hold). Ninety seconds of driver listing per query, 200 queries a day, 30 days a month, comes out to about 150 person-hours a month of engineers waiting on a spinner. At $100/hr fully loaded that's roughly $15,000. Storage isn't free per request either. About 5 billion object requests a month at S3 GET pricing is around $2,000. Add analysts keeping clusters alive on slow queries and that's another $500 to $1,500. And none of this touches Z-ordering, which is basically dead weight at this file size anyway, since data skipping relies on min/max stats per file and a 750 KB file with ~5,000 rows covers maybe a few seconds of event_date range. There's nothing for clustering to work with. A one-time OPTIMIZE on this table runs maybe $500 in compute. Running OPTIMIZE and walking away is the right instinct, execution is usually where it goes wrong. On 847,000 files it'll saturate network I/O, spike DBU costs, and fight with concurrent reads if it's on the wrong cluster. Run it on a jobs cluster, not all-purpose, and scope it: OPTIMIZE catalog.bronze.events WHERE event_date >= current_date() - INTERVAL 7 DAYS. Compacting 18 months of history every night is just burning money. OPTIMIZE fixes file size. Z-ORDER co-locates rows on the columns you filter by, but it degrades as new unordered data comes in. Liquid Clustering (DBR 13.3+) keeps up with that incrementally. Define CLUSTER BY (customer_id, event_date) and a regular OPTIMIZE handles compaction and reclustering at the same time. For streaming tables specifically, enable auto-compaction and stop thinking about it. If you want to check your own tables: DESCRIBE DETAIL, look at numFiles, divide size by file count. Under 32 MB average and you've probably got the same problem. Wrote a short ebook covering this along with some other production Databricks mistakes we hit, silent data corruption, Spark UI triage, medallion anti-patterns, Delta recovery. Happy to answer questions here. submitted by /u/merge_key_skew [link] [comments]

Community Bricktalk Alert | Real-Time Data & AI: Tripwise Demo

Databricks workspace entitlement changes

Databricks workspace access control changes are now being enforced. What's coming? - Azure Databricks | Microsoft Learn Migrate workspace entitlement control - Azure Databricks | Microsoft Learn As of September 14, 2026 , the built-in users and admins system groups behave differently: - users will have no entitelments - admins will have all workspace entitlements - both groups’ entitlements are locked - new principals must be granted entitlements explicitly - users and admins can no longer be nested in other groups Existing access is preserved through a workspace-local migration group: users-clone- If you manage Databricks through Terraform, SCIM APIs, or custom automation, check your workflows now: - Target standard account groups, not system groups - Remove unsupported group nesting - Make sure SCIM sync does not delete the users-clone-* migration group https://preview.redd.it/rx9hw7h25gph1.png?width=668&format=png&auto=webp&s=83a5a4853ec9b19824dd43cbde0596f12a4b0d47 submitted by /u/szymon_dybczak [link] [comments]

Unity Catalog: Pros and Cons

Apache Iceberg won the open table format war when Databricks acquired Tabular, followed by its subsequent adoption across the industry. Then, the catalog war began. In a lakehouse, storing data in object storage and using the Apache Iceberg format is only part of the story. You also need a catalog that helps lakehouse query engines like Spark, Flink, or RisingWave discover tables, manage metadata, enforce access control, and work with governed data across different systems. That is where Unity Catalog comes in. submitted by /u/Low_Brilliant_2597 [link] [comments]

Data and App Releases

Hi, i have a lot of data products which is one part of the project, then we also have a databricks app as AppKit. I was wondering, how do u manage releases/changes when two different bundles and two different responsibilities live in the same place. We use trunk-based development, local IDE for `mode: development`, then staging as pre merge to main, then main is where we create GitHub Release which triggers deployment to production. I think this is pretty standard. But now when whole application comes into the place, we have app/ where we have defined our AppKit template. Should i try to separate lifecycles and deployments fully? To separate GitHub Releases, or i should just let it be inter-winded, but then changes to the app could end up in the deployment of the changes to the data products etc... Right now, i would be using separate releases-draft gh actions which creates draft github releases, and when we are ready to merge, we publish release. For data products we use calendar versioning, for app smth like app-semvar. What is your experience with setup like this? submitted by /u/ptab0211 [link] [comments]

DATA ANALYTCS PROJECT

submitted by /u/Connect_Orchid8513 [link] [comments]

Show HN: Like LeetCode, but multi-file and multi-step

On LeetCode you write one function, it passes, and you're done. Most AI labs such as Anthropic and OpenAI, as well as some big tech companies such as Meta, Snowflake and Databricks, don't stop in one function and instead ask multi-step questions that are closer to real-world problems. Say you're building a key-value store. Step 1 is get, set, delete, count. Once your tests pass, Step 2 unlocks: expiring keys. Then nested transactions. Then atomic batches. I built this because I couldn't find a good place to practice for multi-step, multi-file questions. The questions are based on ones people have reported from those loops. Check it out at https://loopprep.dev. There are 24 questions, 3 free, and no signup to try them. Happy to answer questions. --- top comments --- [MiroslavPokorny] Writing code in a browser like this or leet code is wrong on so many levels its not funny.

Compatibility Mode for AWS Redshift

Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?

Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ⁠ai_query⁠ ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]

Context Engineering, Explained.

submitted by /u/ConstantNo2668 [link] [comments]

Subject: Possible incorrect answer key — Evaluating AI Agents assessment (Generative AI Engineer)

RAG working on Notebook but not on Playground / Agent

Last week I built my first simple RAG to understand the fundamentals (chunking, embedding, etc). It worked great, both on Notebook + Playground. Now, I'm working on another RAG, but this time using ai_parse_document + ai_prep_search as this seems to be the new way to build RAGs in Databricks. So, the chunking part works, I made sure to have only one AI Search endpoint on my workspace (I'm on free edition). The thing is, when I try using the AI index in Playground, and ask questions related to my documents, it says it doesn't have the information i'm asking for. However, when I try the same on a Notebook, it works perfectly. submitted by /u/Relative-Cucumber770 [link] [comments]

I Built an AI-Powered Data Pipeline Generator for Databricks — Here Is What Happened

SDP(ex DLT) ou dbt ?

Bonjour à tous Je voudrais savoir si y’en ici qui ont déjà utilisé SDP(ex DLT) ou dbt dans databricks. Comment est ce que vous intégrez ces outils dans vos workflow ? Databricks native avec SDP ou dbt ou les deux ? Et pour ceux qui ont déjà utilisé les deux outils lequel préférez vous et pourquoi ? submitted by /u/almightysosa888 [link] [comments]

Private Network Gateway

Private Network Gateway is one of the year's biggest network innovations. Serverless can now be part of your VNET! more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?

Non-deterministic ROW_NUMBER() results across Unity Catalog environments

The Missing Correlation Layer in Databricks ModelOps | Building an AI Health Control Plane for DBx

Ho creato un filtro deterministico per record JSON su Apify: accetto feedback.

submitted by /u/LucaDataTools [link] [comments]

Just ask Genie

"Just ask Genie." You hear this phrase more and more often when questions about Databricks arise. It frequently appears in Academy materials, too. AI reduces routine work it’s convenient, and we get used to it. But do we stop thinking in the process? The problem arises when a person no longer understands why a given answer is correct. Does AI really free us from the need to know the details, or does it actually make fundamental knowledge even more important? submitted by /u/Significant-Guest-14 [link] [comments]

I’m a certified associate data engineer. What’s next?

I’ve been working as a “data engineer” in Azure Databricks for a while, but I work on a team where my scope is extremely limited to silver/gold work. What studies can do I do next to continue developing? I’m thinking of diving into the azure certs submitted by /u/WeirdAnswerAccount [link] [comments]

UC Delta API: managed-table READ_WRITE credentials rejected despite explicit User-Agent

De role evolution - where are things going?

submitted by /u/Cultural-Reserve-259 [link] [comments]

Writing nice unit tests is impossible

First of all we have a lot of classes that use the DatabrickSession import, which makes unit testing impossible and whenever I have the "normal" spark import and test it, then it looks absolutely ugly. Would it help to have the schema as json? I use json schema to read for transformations. If I want to test without json its horrible, do you guys test with json schema files? For example this is just the output, then I would need 2x this because of the input and the test is unreadable (according to my senior BUT WHAT AM I SUPPOSED TO DO ): schema = """ id INT, items ARRAY fortnite: STRING, babies: INT, moreStuff: MAP >> """ expected = spark.createDataFrame( [ { "id": 1, "items": [ { "fortnite": "ABC", "babies": 2, "moreStuff": { "size": "L", "color": "red", }, } ], } ], schema=schema, ) submitted by /u/Similar-Bug-350 [link] [comments]

Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs

igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?

Task Success is not Routing Health: a Databricks Smart Routing Case Study

Build with Replit Against Managed Postgres in Databricks Apps

DevOps Essentials for Data Engineering: Lab does not match Videos

Databricks Solutions Architect Vibe Coding Round

Has anyone recently taken the databricks Solution Architect interviews? I have a vibe coding round with databricks in next week. I'd appreciate any guidance! Thank you 😊 submitted by /u/wilysarah [link] [comments]

How do you parse an xml that's in string format?

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this: .... ... ... Regex is not an option. Substring is not an option. Because it's so messed up it's not consistent. I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql Please, help you mate. submitted by /u/pineapple_brownies [link] [comments]

Choosing the right format, explained.

submitted by /u/ConstantNo2668 [link] [comments]

Jobs and Runs UX is Frustrating

I'm not new to Spark, but I'm pretty new to the Databricks platform. I am finding that the UX for monitoring jobs and runs is very rigid, and doesn't present my workloads as I would expect. Here is one simple example. If I submit a run with the "jobs/runs/submit" API then I can provide a custom and ad-hoc "run_name" that appears in the management console called "Runs". This is good. But if I submit a run that references a pre-existing job (using the "jobs/run-now" API) then there is NOT a way to provide a custom "run_name" that will be displayed in the databricks console. The only name that can be shown is the job's name. There are other things that don't seem right either. If I enter custom "tags" on my jobs, then I will be able to use the tags to filter on the Jobs list. But when I click on the Runs list, I can't filter on those same "tags" anymore. IMO, those tags are just as useful on BOTH screens. Another example - the UX doesn't allow me to show more than 20 completed runs at a time. I have to click the Next/Previous button to find runs. Paging thru a long list of runs is a really painful experience. I'm also a user of Microsoft Fabric. I once thought that the "Monitor" console of Fabric was pretty unfriendly ... but now that I'm in Databricks I realize that I'd much rather use their endless scrolling UX design, than having to spam-click the Next/Previous buttons. Even the HDInsight-yarnui allowed me to navigate my workloads more easily than I can in databricks; and that UX is a decade old by now! Is there a different UX experience for Jobs and Runs that I'm missing? Maybe a VS code extension in the community or something like that? Any tips would be appreciated. submitted by /u/SmallAd3697 [link] [comments]

Why External Secrets in Unity Catalog Matter

Lakeflow Connect SQL Server Connector

I recently enabled Lakeflow Connect (lfc) on the source database - the issue is, some of the tables in the source database (managed by another team) does NOT have a primary key (which means that in lfc, a __databricks_id is used to identify a unique record). Thus, the DBAs enabled CDC on the source database. However, when I ingested the data into DBX using the Lakeflow Connect Managed SQL Server Connector, one of the tables in the source database had duplicate records (two or more records with the same value across all columns). This caused my Lakeflow Connect pipeline to break. Any ideas on how to fix this? (Other than dropping duplicate records in the source DB and implementing a unique constraint on the source DB)? I was wondering if there is a specific setting in Lakeflow Connect that I can toggle that I'm missing. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Delta Executor: Things to Check Before Using a Roblox Tool

Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics

Broken x-axis sort order in combo chart

Meet SDP Rewind: An undo button for your ETL pipelines

Announcing Databricks AppQuest: Hands-On Guidance and Cash Prizes for APJ Developers

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

CUSTOMER STORY | Siemens Healthineers modernizes MRI scanner data on Databricks

Best Practice for Handling Schema Evolution with Auto Loader in Production?

Announcement | Autoscaling Lakebase Postgres

Changing Root Metastore Location

CSV export from AI/BI Dashboard table widget does not preserve visual column order

Getting started with Delta Lake basics

Storage Credential creation fails with "Access Connector ... could not be found"

Free Trial – “Daily limit for workspace creation” error

External secrets in Unity Catalog is in Beta, and it replaces Key Vault-backed secret scopes

This is the Databricks release I have been waiting for. Unity Catalog schemas can now hold external secrets, such as Azure Key Vault, and that will change how we manage and utilize secrets in our Databricks projects. On most of our engagements the secrets of record already live in Azure Key Vault, so we wire up a Key Vault-backed secret scope and move on. It works, but it is a workspace-level object from the pre-Unity Catalog era: configured per workspace, permissions managed through a separate secret ACL API, a flat scope/key namespace, and invisible to the governance model everything else on the platform now runs on. Read more: https://www.linkedin.com/posts/cenh_databricks-azure-unitycatalog-ugcPost-7504125176993800192-W9on/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]

How do you test whether a retriever stops too early?

Databricks Unity Catalog Explained | Full Governance Guide (Access Contr...

submitted by /u/macxima [link] [comments]

UC secrets in Key Vault

Secrets in Unity Catalog is a great feature introduced a few weeks ago, but since then, everyone has been asking to use Azure Key Vault as a secrets backend. Thanks to rapid development, we can now link our schema to Azure Key Vault; UC will read secrets as UC secrets, and permission management will be through Unity Catalog. In that scenario, you insert/update secrets in Azure Key Vault, but read/reference and grants can go through UC. more news https://databrickster.medium.com/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

Building for Failure: Implementing Data Quality Firewalls in Petabyte-Scale Medallion Architectures

Azure AI Foundry + Databricks Architecture | Deploy Genie Agent with DAB...

Azure AI Foundry Databricks architecture, Deploy Genie Agent with DABs, Databricks Genie Agent, Azure Databricks Genie Space, how to deploy genie agent with declarative automation bundles, azure ai foundry + databricks integration, fully operating genie architecture databricks, databricks unity catalog genie agent, azure databricks bronze silver gold architecture, agent to agent nlq databricks, databricks spark python sql delta lake unity catalog, production ready genie agent deployment, databricks vector search index genie, microsoft purview databricks governance submitted by /u/macxima [link] [comments]

Designing an Effective "Quarantine" Pattern for Failed DLT Expectations

How to create monotonic function to incrementally add obj_id for datasource in pyspark

The Future of Iceberg Isn't One Engine. It's an open Control Plane with many engines.

submitted by /u/codingdecently [link] [comments]

Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes

Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Controlling allow/ask/deny lists for predelivered MCP connectors

Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines

Where Omnigent Fits Alongside ChatGPT Work, Claude Cowork and Cursor Projects

Do we still need fact and dimension tables in the Gold layer?

Data engineers traditionally modeled Gold layers using fact and dimension tables, partly because storage and compute were expensive. But with modern cloud data platforms, storage and compute are much cheaper and more scalable. So I’m curious: what does your Gold layer actually look like today? Are you still using a traditional star schema (facts + dimensions), or have you moved toward wider, denormalized tables / business-oriented models? And more importantly, why? submitted by /u/almightysosa888 [link] [comments]

Migrated our reproting layer to databricks and access control turned into a project of its own

The actual data movement into databricks did go fine (unity catalogue made lineage way easier to see than the older setups). But what we didn't expect was the time that went into access control, once everything was centralized instead of scattered across separate warehouses with their own permissions, data that used to be siloed was suddenly way visible to more people by default, which was found out when a couple of teams noticed they could see data which they probably shouldn't. Spent almost as much time on catalog level permissions/row filtering as on the actual pipeline work. Is this normal for a databricks migration or did we happen to have an unusually messy access management? submitted by /u/BrownAnclourne [link] [comments]

What Data Engineers Need To Know About Delta Lake 4.3

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog. submitted by /u/Lenkz [link] [comments]

Google Drive connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Google Drive is now generally available ! It’s now easier than ever to ingest structured and unstructured files from Google Drive into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich Google Drive metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at Google Drive URLs. https://preview.redd.it/j65yaaft3uoh1.png?width=2048&format=png&auto=webp&s=0f9f13cc63572b991232a8fa617aa1fe6697369c Link to public docs + references: Google Drive managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Common workloads include: Loading Google Sheets, Excels, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Google Docs, Google Slides, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. Examples of using the Spark + SQL APIs: Read an Excel sheet from Google Drive with spark.read : ​ df = (spark.read .format("excel") .option("databricks.connection", "my_gdrive_conn") .load("https://docs.google.com/spreadsheets/d/9k8j7i6f...")) Ingest unstructured documents + PDFs from a Google Drive URL with read_files , then easily parse them using ai_parse_document : ​ CREATE OR REFRESH STREAMING TABLE gdrive_documents_table AS SELECT *, "_metadata" FROM STREAM read_files( "https://drive.google.com/drive/folders/1a2b3c4d...", format => "binaryFile", `databricks.connection` => "my_gdrive_conn", pathGlobFilter => "*.{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM gdrive_documents_table; Coming soon: Ingest Google Drive’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are building and let us know if you hit any friction! submitted by /u/BricksterJ [link] [comments]

SharePoint connector in Lakeflow Connect is now generally available (GA)

The Lakeflow Connect connector for Microsoft SharePoint is now generally available! It’s now easier than ever to ingest structured and unstructured files from SharePoint into Delta tables for analytics and AI workloads. You can configure a managed ingestion pipeline through the UI or managed API. Managed pipelines automatically handle incremental processing, automatic retries with exponential backoff for source API rate limits, failure recovery, and provide rich SharePoint metadata. Soon, our managed connectors will also support ingesting SharePoint Lists and per-file permissions metadata. For direct control over ingestion logic, you can also just use the Spark + SQL APIs directly: spark.read , Auto Loader, read_files, or COPY INTO pointed at SharePoint URLs. Common workloads include: Loading Excel, CSV, JSON, and other structured files into Delta tables. Ingesting PDFs, Word documents, PowerPoint files, and images. Parsing documents with ai_parse_document to prepare content for extraction, search, and agents. https://preview.redd.it/89i379aattoh1.png?width=2180&format=png&auto=webp&s=292350dfddfc9bc1a4daf3fc4447394821206054 Link to public docs + references: SharePoint managed connector documentation Spark + SQL APIs and examples Community blog and video tutorial: From PDF to insights Data + AI Summit session: Intelligent Document Processing with Lakeflow Examples of using the Spark + SQL APIs (after first creating a UC connection ) : Read an Excel sheet from SharePoint with spark.read : excel_df = (spark.read .format("excel") .option("databricks.connection", "my_sharepoint_conn") .option("headerRows", 1) .option("dataAddress", "Sheet1!A1:M20") .load(" https://mytenant.sharepoint.com/sites/Finance/Shared%20Documents/Monthly/Report-Oct.xlsx") ) Ingest unstructured documents + PDFs from a SharePoint URL with read_files , then easily parse them using ai_parse_document CREATE OR REFRESH STREAMING TABLE sharepoint_documents_table AS SELECT , "_metadata" FROM STREAM read_files( " https://mytenant.sharepoint.com/sites/Marketing/Shared%20Documents ", format => "binaryFile", databricks.connection => "my_sharepoint_conn", pathGlobFilter => " .{pdf,docx}"); CREATE OR REFRESH STREAMING TABLE documents_parsed AS SELECT *, ai_parse_document(content, map('version', '2.0')) AS parsed_content FROM STREAM sharepoint_documents_table; Coming soon: Ingest SharePoint Lists into Delta tables (coming super super soon) Ingest SharePoint’s per-file permissions and ACL metadata to power permission-aware AI agents, enterprise search, and more. If you try it, share what you are ingesting and where you hit friction! Don't hesitate to ask questions! submitted by /u/BricksterJ [link] [comments]

Unable to Start Serverless Compute in Databricks Free Edition

Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices

Handling New Columns in a Databricks Data Pipeline

Incremental data processing explained!

submitted by /u/ConstantNo2668 [link] [comments]

OAuth M2M (client-credentials) - getting error with github run

Dark mode setting ignored when embedding BI dashboard inside Databricks App

The Replit | Databricks Integration is now GA for building governed apps

submitted by /u/tony-dang [link] [comments]

Databricks metric views to PowerBI?

submitted by /u/FiftyShadesOfBlack [link] [comments]

[Megathread] self promotion

Hey r/databricks , In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules! submitted by /u/AutoModerator [link] [comments]

[Megathread] Certifications and Training

Hey r/databricks , please direct all certification and training posts here. There's upcoming learning festival September 16 - October 14 2026. You can get 50% discount voucher on any certification. Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157 Good luck to everyone on your certification journey! submitted by /u/AutoModerator [link] [comments]

[Megathread] Hiring and Interviewing at Databricks - Advice, Prep, Questions

Hey r/databricks , we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions. submitted by /u/AutoModerator [link] [comments]

🚀 Quest 5 Submission: Intelligent RAG Knowledge-Base & Note-Taking Workspace

Allow Public Network Access - Disabled, I can still access the workspace from public internet

Nexa: Genie is a Minute Away

Smart Routing in Unity AI Gateway: 30%+ Cost Savings on Coding Tasks

Any plans to make externally backed secrets in Unity Catalog enter public preview/GA?

Hi Databricks Team, Seeking your advice on the above. submitted by /u/RazzmatazzLiving1323 [link] [comments]

Metric views materialization failure

Building Deterministic Databricks Genie Agents

How do I read the databricks spark ui? Couldnt find any tutorials specifically for it.I know spark ui a bit.

submitted by /u/Intelligent_Duck_854 [link] [comments]

Databricks Community Contest | Winners of the Genie-Powered App Challenge!

Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation?

Databricks Quest 4-(Deploy Your First Databricks App) success story

Databricks App suddenly unable to load valid Jupyter notebook

Actually understanding Unity Catalog Managed Tables

Looking for Databricks Data Engineers in EU - Fully Remote

I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience. Would be a 6-month initial contract, would be open to a further extension if needed. If this is something you'd be interested in, then comment below. I will ping you. €600-650 per day submitted by /u/Reuben_UMATR [link] [comments]

Agent outside databricks communication with databricks delta table

Materialized Views vs Streaming Tables in Databricks: A Practical Guide for Data Engineers

databricks-langchain PyPI installation stuck for 30+ minutes on multiple Azure Databricks clusters

Announcement | How we eliminated $1 million a year of wasted AI agent spend in one hour

shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log

How are you combining forecasting models for time-series work?

[Discussion] Lakeflow Jobs: How do you use table update triggers?

(databricks product manager here) Curious how people are using table update triggers in production. https://docs.databricks.com/aws/en/jobs/trigger-table-update Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic? For example: Do you use control or checkpoint tables to signal that an upstream workflow has finished? Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work? Do you put that logic in the trigger itself, or in a separate workflow/job? What has worked well and what has been difficult to reason about or debug? Any other suggestions or feature requests relating to Table Update Triggers? I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach. Edit: how many folks still use control tables instead of data tables with these triggers? Thank you 🙏 https://preview.redd.it/munx0r2u4ooh1.png?width=660&format=png&auto=webp&s=80e6b80a62cf77870bf444e7634d1ee7412feee3 submitted by /u/saad-the-engineer [link] [comments]

Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations

Databricks Support -0100505- Exam Suspended because non compliance requirements set by test sponsor

Managing Schema Drift and Evolution in Spark Declarative Pipelines (SDP)

No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring

https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]

Before you make your pipeline “near real-time”, check where the latency actually is

Here is a common pattern: a job polls a queue every 20 minutes, fans the payload out to 60 to 80 Bronze tables using MERGE operations, and takes 16 minutes to complete. When leadership asks for “near real-time,” the default response is to drop the polling interval to one minute. That approach fails because of simple arithmetic. Worst-case latency can be roughly the polling interval plus the batch duration, which puts total time at 36 minutes. Setting the trigger to 1 minute while the batch takes 16 minutes won’t give you 1-minute latency. It can instead create queued runs and additional contention between jobs. You need to optimize the batch duration first. In wide fan-out architectures, two bottlenecks can cause significant delays: MERGE operations running on empty targets. If a batch only updates 6 out of 82 tables, the other 76 MERGE operations are unnecessary work. Partition the payload first, check which targets actually received rows, and skip empty writes. Sequential writes. The 82 tables are independent, so processing them one by one in a driver loop can make the batch duration approach the sum of the individual write latencies instead of being closer to the slowest write. Where appropriate, independent writes can be processed concurrently. Address these two points first to reduce batch duration before shortening the trigger interval. Once that is done, re-evaluate whether you actually need a streaming architecture. For anyone being pushed to deliver “real-time” processing: what latency requirement did the business actually need once it was clearly defined? In practice, teams sometimes ask for seconds when minutes would actually meet the requirement. submitted by /u/AbilyticsEng [link] [comments]

Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]

Looking for buddy

Hey guys Im from Hyderabad, India . A databricks dataengineer here. As we have event on 7th Oct 2026 in Mumbai, im planning to visit it. Who else are joining. Lets have some good connections 😌 submitted by /u/PrinceShahil6 [link] [comments]

Resolution of Missing Certificates in Accredible

The deployment decalogue

I am an ML engineer, but I come from a software engineering background: years of full-stack work, with heavy DevOps and Terraform experience. I come from teams that deploy to production five times a day with real continuous deployment. And honestly? Pressing the button still feels weird sometimes. Every engineer knows that feeling, no matter how good the safety net is. So I wrote down the list that settles it. Ten commandments, one flow, written with data scientists and ML teams in mind, but it works for batch jobs, realtime inference, and LLMs alike. Answer honestly, and if all ten are true, you can ship to production anytime, in any form or way. submitted by /u/SuspiciousPavement [link] [comments]

Enterprise workspace blocked by Databricks-set rate limit of 0 on all models (AWS Marketplace)

BOOTSTRAP_TIMEOUT on cluster start (Southeast Asia)

Databricks AppQuest - Dev kit !

Are Unscored Questions Really Unscored?

Unity Catalog Open Source in Name Only (UCOSINO)

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above). Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error. But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.) It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github! I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase. It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public. submitted by /u/SmallAd3697 [link] [comments]

datatf: Automate importing Databricks workspace into Terraform

DataTf , bring an existing Databricks Workspace into Terraform (IaC). generates dynamic terraform.tfvars + import code built on the Databricks Go SDK compatible with Terraform or OpenTofu compatible with Databricks Omnigent, Claude Code, opencode, OpenAI Codex and others native support for Azure, with GCP and AWS support upcoming Disclosure: I am the Author/Founder, 536 Technologies. submitted by /u/536tech [link] [comments]

Has anyone tried the new Databricks AI/BI feature?

I recently came across Databricks AI/BI and was curious to know how people are finding it. It looks like Databricks is trying to bring BI and analytics more directly into the Databricks platform, with dashboards and Genie for asking questions in natural language. Has anyone actually tried AI/BI in a real project? How is it compared to Power BI or Tableau from your experience? Is it good enough for regular BI use cases, or is it still better to use a separate BI tool? Would like to know your experience, especially if you have used both. submitted by /u/Bhanuprakash_1947 [link] [comments]

How to organize your notebook tabs?

It was a real pain, but now, with a few tricks, you can manage them better. First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook. If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like. more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a submitted by /u/hubert-dudek [link] [comments]

A single Databricks Architect Champion can potentially affect a company’s Partner Tier

There are 1,600+ Databricks Partner Champions worldwide and around 1,795 partners listed in the public directory. But Champions are distributed very unevenly. Some large partners have dozens. Others may have only enough to meet their current tier requirements. So if one Architect Champion leaves, the company could potentially fall below one of the requirements for its Partner Tier. That makes Champion status more than just another badge — it can have real value for the employer. For Databricks architects working at partners, I’d definitely put Partner Champion on the career roadmap. Do your companies maintain a buffer of Champions, or just the minimum required? submitted by /u/Significant-Guest-14 [link] [comments]

Reissue Voucher

Cost-optimized way to reflect source DB changes in Silver in <1 minute?

Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute. Currently, the flow looks like this: Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode. The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow. This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need. However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that. My question is : is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer? Maybe Lakeflow Connect or some custom process? Extra notes: - I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently. - I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute. - I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale. submitted by /u/CyberEnzo [link] [comments]

Want to become a Forward Deployed Engineer?

We’re running a hands-on Forward Deployed Engineering (FDE) workshop focused on that gap: turning ambiguous customer problems into secure, governed, production-ready AI deployments. You’ll work through a realistic 90-day AI-agent deployment for a regulated customer, covering: evaluation and reliability security and governance rollout and deployment risks stakeholder alignment a CISO hot seat The workshop is led by Keith Bourne, Forward Deployed AI Engineer at Tribe AI, and Tanya Dixit, Forward Deployed Engineer at Google. There’s also a practical FDE career-path assessment to help you identify your strengths and gaps, explore relevant employers, and build a personalized 90-day career plan. Here is the link for futher details: https://luma.com/fde-workshop?utm_source=career submitted by /u/kunal_packtpub [link] [comments]

Genie Agent's idea of "the Midwest" includes Kentucky. Ours doesn't. Notes from [8 months] of Genie Spaces in prod.

Genie Space has been live for our sales folks for 8 months, maybe 500+ regular users. Short version of what I've learned, since everything I read before setting one up was either a demo or an argument about whether analysts are getting replaced. The failure mode isn't an error. It's a number that's slightly wrong and totally believable. Someone asked how the Midwest was doing, the number looked fine, sat in a deck for weeks. Genie's Midwest includes Kentucky. Our territory map doesn't. You can't catch that by looking at the output - you catch it when finance does. Four things you're configuring, roughly in order of how much they've mattered: Column comments. Free, and the biggest lever by far. Genie reads COMMENT metadata before writing SQL. No comment and segment is just a word - it has no idea whether your values are Enterprise/Mid-Market/SMB or something else, so it guesses. COMMENT ON COLUMN vw_sales_summary.segment IS 'Customer tier: Enterprise (>$1M ARR), Mid-Market ($100K-$1M ARR), SMB ( One pre-joined view, not raw tables. I did raw tables first. Every join it has to figure out is a coin flip. Also, put your test-data filter in the view - then every question anyone ever asks inherits it and you're not trusting the model to remember. SQL expressions. Register a named metric with your SQL and it uses yours instead of inventing one. Ask ten people what an "active customer" is and you'll get eleven answers; this is the box where you settle it. Name them how people talk - "Active Customers" matches, cnt_dist_cust_qtd never will. Example Q&A pairs. Nothing gets retrained, they just sit in context when something similar comes in. The shape travels further than I expected — registered revenue-by-category with a cancelled-order exclusion, and a Q2 question a month later inherited the exclusion in a query I never wrote. Two things from the instructions box worth stealing. One, tell it to ask instead of guessing when the time period is unclear - people trust it more when it occasionally asks. Two, ours has a rule about test customers with a TST_ prefix, whose orders carry real statuses so the status filter misses them entirely. Everyone on the team knew that. Nobody had ever written it down. Curious what other people have ended up putting in their instructions box. Assume everybody hits their own Kentucky eventually. (Here is the longer version with more SQL is on SQLServerCentral, it's mine https://www.sqlservercentral.com/articles/databricks-genie-spaces-for-sql-analysts-natural-language-querying-without-leaving-your-data-platform but the above is the useful part) submitted by /u/mehulbhuva [link] [comments]

Databricks Dashboard Embedded Ask Genie

Why we didn't build one big Genie Room

Can't access abfss data in azure databricks when providing shared key (fighting UC?)

Do UC table tags propagate to billing for Predictive Optimization and Data Quality Monitoring?

Introducing Stream-Stream Join Support in Apache Spark Real-Time Mode

CUSTOMER STORY | Rippling powers AI-driven GTM with Genie Agents on Databricks

Multiple gateway pipeline for same database

How are you separating dev, staging and prod in Unity Catalog without duplicating everything?

Data migration from teradata to databricks

I joined a new company recently and got a migration project here, they are migrating from teradata on prem to databricks, I have never done any migration in the past can anyone suggest some helpful yt videos or any other knowledge source? submitted by /u/WarPowerful740 [link] [comments]

Databricks Secrets

Databricks Account locked Issue

Learn Databricks Lakeflow | Ingest, Orchestrate, and Build pipelines on one platform.

Need some advice on Snowflake vs Databricks

submitted by /u/Ok_Independent_343 [link] [comments]

How to automate downloading files from Databricks to a local machine without PATs or CLI?

Hey everyone, Looking for some advice on automating a workflow in a pretty locked-down corporate environment. Context: Large enterprise with strict IT security and governance. Databricks was recently rolled out as our cloud data hub. The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks. I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download." Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions? Thanks! submitted by /u/Firm_Yogurtcloset835 [link] [comments]

Can I run jobs continuously without interruption on a permanent free (Community Edition) account?

What to learn? AWS databrics or Azure Databrics

I have experience in AWS and I want to learn Databricks now From future perspective what I need to learn Databrics with AWS or Databricks with Azure? I can see there are lot of openings related to Azure with Databricks Can anyone plz help me submitted by /u/mali_sagar [link] [comments]

What's new in Genie One - August 2026

submitted by /u/Youssef_Mrini [link] [comments]

Issue Regarding Missing Badge on Profile: DAIS24 Attendee

Databricks Dashboard Pivot Table: default collapsed state?

Actor for exporting database data into Datasets

submitted by /u/Hayder_Germany [link] [comments]

DELETE Removed the Customer Record—but Did It Remove the Data?

CosmosGenie — Your Universe, Answered (Genie-Powered App Challenge )

Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"

Harmonizing Informatica Governance Policies Across Databricks and Power BI Import Mode

Could Databricks app support the integration of Genei Room and Power BI?

Expanding Genie Agents: Deep analysis, file reasoning, and more

How are you attributing serverless costs back to individual jobs and teams?

Now that Unity Catalog manages Iceberg natively, is anyone actually switching?

AI/BI Dashboards

🌟 Community Pulse: Your Weekly Roundup! August 31 – September 06, 2026

Apache Iceberg Table Cleanup: A Production Guide

A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle. submitted by /u/codingdecently [link] [comments]

Exploring the Databricks Application Development Ecosystem

Databrick Oauth federation

Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project

Could Databricks app support the integration of Genei Room and Power BI?