Databricks SQL

Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?

Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ⁠ai_query⁠ ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]

How do you parse an xml that's in string format?

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this: .... ... ... Regex is not an option. Substring is not an option. Because it's so messed up it's not consistent. I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql Please, help you mate. submitted by /u/pineapple_brownies [link] [comments]

No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring

https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]

Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]

Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand. My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses. Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency. When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs. Would love to know what you guys think. submitted by /u/Puzzled-Mail-9092 [link] [comments]

What is a Warehouse in Databricks SQL?

Query works in Databricks SQL but fails through JDBC

I’ve noticed cases where a query runs successfully in the Databricks SQL editor but fails when the exact same query is executed through a JDBC-based application or data quality tool. Has anyone run into this? Was the issue related to query wrapping, session settings, SQL dialect differences, or JDBC driver behavior? How do you usually troubleshoot these cases? submitted by /u/No_Ambition8323 [link] [comments]

What type of compute do you guys use in Databricks?

Hey everyone, I've been using Databricks for a while, but I'm still curious about how other teams decide which compute option yo use. There are so many choices now - serverless,job clusters, all-purpose clusters, SQL warehouses, etc. - and sometimes it's not really obvious which one makes the most sense. And when do you go with serverless vs a normal cluster? I'm also wondering if people are mainly choosing based on cost, performance, startup time, or just what their team is already comfortable with. Would be interested to hear what you're using in your projects and what made you choose it. Especially if you've switched from one type of compute to another and noticed a real difference. submitted by /u/Delulu62134 [link] [comments]

Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]

Moving ~150 Tableau dashboards to Databricks AI/BI - anyone done this at scale?

We are considering moving from Tableau to Databricks AI/BI Dashboards and wanted to see if anyone here has made a similar move. We currently have around 150 Tableau dashboards used by 1,000+ users, with a mix of technical and non-technical users. Most of the underlying data already lives in Delta tables, so naturally we're looking at whether it makes sense to consolidate and use Databricks for the dashboarding layer as well. For those who have done this, how has the experience been? Anything you wish you knew before migrating? Any major limitations compared to Tableau, especially around visualization, performance, or the experience for non-technical users? I'm also curious about cost at scale. With 1,000+ users potentially hitting dashboards, did SQL warehouse usage become a concern? Would you feel comfortable replacing Tableau completely with AI/BI Dashboards today, or do you still find yourself needing Tableau for certain use cases? Would really appreciate hearing about real-world experiences. submitted by /u/Immediate_Bus5250 [link] [comments]

databricks SQL UDF in select statement

"List columns" statements when querying Databricks via ODBC

I've got an old .NET app running on Windows Server. The app submits queries to a serverless SQL Warehouse. I've noticed that each query ( SELECT statement) submitted by the app results in three statements in the SQL Warehouse. One statement containing the query submitted by the app, and two statements that look like Listing columns 'catalog : foo, schemaPattern : foo, tablePattern : resultssettable, columnName : null' . The query might take 200ms to execute. The "Listing columns" statements take about 90ms each to execute. In aggregate, these "Listing columns" statements double the duration of queries sent to this warehouse. How do I get rid of these freak'n "Listing columns" statements? Here is what I've done/tried: Tables in the query are full qualified with catalog and schema. Query specifies the names of the columns to retrieve (i.e., not a SELECT * FROM ... ). I have Use Native Query enabled in the DSN. I have Fast SQLPrepare enabled in the DSN. Enabling Ignore Tables Metadata from All Schemas in the DSN makes no difference. I configured the Databricks ODBC driver to pool connections. I have the 64-bit Databricks ODBC driver installed, version 2.11.00.1005. I don't have access to the .NET apps source. I have been using the following PowerShell script to recreate the issue and test configuration changes: $dsn = "DSN=Test Connection" $sql = @" SELECT col1 , col2 , ... FROM dev.foo.bar WHERE whatever = 1; "@ $conn = New-Object System.Data.Odbc.OdbcConnection $dsn try { $conn.Open() $cmd = $null $cmd = $conn.CreateCommand() $cmd.CommandText = $Sql $cmd.CommandTimeout = 30 $adapter = New-Object System.Data.Odbc.OdbcDataAdapter $cmd $adapter.MissingSchemaAction = [System.Data.MissingSchemaAction]::AddWithKey $table = New-Object System.Data.DataTable "foobar" [void] $adapter.Fill($table) return $table.Rows.Count } finally { if ($reader) { $reader.Close() } if ($cmd) { $cmd.Dispose() } } I feel like I missing […truncated]

Databricks SQL Alert - Scheduling Questions

Databricks SQL Alert — Question Set Help needed, I created a SQL alert in databricks to notify me when a new entry is added to a dataset/table within the last 24 hours. SQL SELECT COUNT(*) AS new_ FROM catalog.schema.table WHERE xyz IN ('a', 'b', 'c', 'd') AND datetime >= CURRENT_TIMESTAMP() - INTERVAL 24 HOURS; Within the Alerts panel, I set the condition as follows: First row Column name: new_ Operator: > Static value: 0 I then added my email address in the Notifications section and set "When alerting, notify" to Always . I then selected View Alert , where I created a schedule — let's say every day at 9:00 AM . Questions a) How exactly does the schedule work here? I am asking because I selected "Always" under the notification settings, but I have also created a schedule to run every day at 9:00 AM. What is the relationship between these two settings? b) Let's suppose I build this alert and create the schedule as described above. Person B (sales dept rep) also subscribes to this alert. I do not want Person B to receive an alert unless the new record belongs to the Sales department. For example, if a new record is added that belongs to a department other than Sales, and the scheduled alert runs at 9:00 AM, will Person B still receive an email notification? Thanks in advance for any clarification! submitted by /u/SquareLong2523 [link] [comments]

Databricks Dashboard - Scheduling Questions

( FYI, I did write the initial draft of this message by myself, and then later asked AI to proof read it and later pasted the same here, Apologies in advance!) I recently developed and published a Databricks dashboard for a personal project, and I want to learn how the scheduling feature works. For context, I have read the Databricks documentation, Manage scheduled dashboard updates and subscriptions | Databricks on AWS , but I couldn't find answers to the questions below. Databricks Dashboard — Question Set 1 I published the Databricks dashboard using Individual Data Permissions . I then added another person as a participant, created a schedule, and subscribed to the schedule from both accounts. Let's call them Person A (publisher + subscriber) and Person B (subscriber). Questions a) Assume Person B works in the Sales department and should only be able to see Sales data. Since the schedule was created by Person A, when the scheduled run occurs and Person B receives the email notification they subscribed to, will the PDF attachment contain all departments' data , or will it contain only the data Person B is permitted to see? b) After the scheduled run, if Person B opens the dashboard using the link in the email notification, or accesses the dashboard directly through Databricks, what data will they see? Will they see all department data , or only the data they are permitted to see based on their individual data permissions? My understanding is that when Person B visits the dashboard, it does not automatically refresh just because a scheduled dashboard update has occurred. I may be misunderstanding how this works, so I'd appreciate some clarification. Databricks SQL Alert — Question Set 2 I also created a SQL alert to notify me when a new entry is added to a dataset within the last 24 hours. Please note: The screenshot below is not from my actual example. I am only including it so that new developers like myself can relate to what I am referring to. enter image descr […truncated]

Tagging individual ai_query() calls in SQL?

Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps. We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step. From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL. Has anyone found a workaround, or is moving the calls out of SQL currently the only option? Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc submitted by /u/hulioshort [link] [comments]

Load Testing Databricks SQL Warehouses with JMeter — Part 3: Running and Analyzing Results

Load Testing Databricks SQL Warehouses with JMeter — Part 2: Concurrency and the Test Plan

Load Testing Databricks SQL Warehouses with JMeter — Part 1: Inputs and Configuration

Serve Tableau Reports Directly from Databricks SQL

Understanding EXPLAIN FORMATTED in Databricks SQL

DBSQL MCP output limit

Querying Metric Views via Classic/Pro SQL Warehouses

Incremental REPLACE WHERE Flows Brings Targeted Refreshes to SDP and DBSQL

Restrict certain queries on SQL Warehouse

SQL Warehouse stuck on "Cluster Start-up Delayed"

Databricks SQL connection becomes stale in long-running app

Databricks SQL Just Dropped Some Massive Engine Upgrades for Data Engineers

Debugging a Databricks Federated Query

Hi, I came across an interesting question on the Databricks Community, and I thought I would share my findings in case someone else runs into the same issue. Maybe this will help save some time debugging. The setup was as follows: * SQL Server was registered as a federated connection in Unity Catalog. * Queries were executed directly from Databricks against federated SQL Server tables. * The user who created the thread executed a query using standard date functions such as `YEAR()` and `MONTH()` against a federated table. Example of query that was executed: SELECT MONTH(order_date) AS order_month, YEAR(order_date) AS order_year, COUNT(*) AS total_orders FROM federated_db.sales.orders GROUP BY MONTH(order_date), YEAR(order_date) However, the query failed with the following error: `com.microsoft.sqlserver.jdbc.SQLServerException: 'EXTRACT' is not a recognized built-in function name. at` So, what is happening here? At first glance, this looks confusing because SQL Server does support the `YEAR()` and `MONTH()` functions. As it turns out, in Databricks SQL, `YEAR()` and `MONTH()` are synonyms for the ANSI **EXTRACT()** function. https://preview.redd.it/5grz6yvu7n3h1.png?width=765&format=png&auto=webp&s=19cb791a079ddbbe2eb696f77d1693028d0bd722 That is where the problem occurs: SQL Server does not support the ANSI `EXTRACT()` syntax, so the federated query fails. To work around this, you can try rewriting the query in a way that is compatible with the federated execution path. Alternatively, you can use the **remote\_query()** function, which allows you to run SQL directly against the external database using the native SQL syntax of the remote system. Orignal thread: [Re: Extract SQL function in SQL Server federated d... - Databricks Community - 157392](https://community.databricks.com/t5/data-engineering/extract-sql-function-in-sql-server-federated-database/m-p/157397#M54544)

Why Databricks Genie returns MessageStatus.FAILED in App API (but works fine in UI) — solved

Hey fellow Databricks builders, Our team at **Enqurious** just wrapped a hackathon building a full medallion architecture (Bronze → Silver → Gold) with Genie AI integration for natural language querying. We hit a wall that cost us 6+ hours — sharing the full fix so you don't have to. # What was the setup? * Gold layer tables in Unity Catalog * Genie Space configured and working in UI * Databricks App deployed successfully # What causes MessageStatus.FAILED in Databricks Genie App API? The root cause is a missing service principal permission chain. The Databricks App runs under a **service principal** — not your user account. Genie working in the UI only proves *your user* has access, not the service principal the App uses at runtime. This is why the same question works in the Genie UI and fails via the App API every time. # What is the complete permission fix? The service principal needs **all five** of these — missing any one causes silent failures: sql -- 1. Catalog level (most commonly missed) GRANT USE CATALOG ON CATALOG your_catalog TO `your-app-service-principal`; -- 2. Schema level GRANT USE SCHEMA ON SCHEMA your_catalog.your_schema TO `your-app-service-principal`; -- 3. Table level — each table individually GRANT SELECT ON TABLE your_catalog.your_schema.table_1 TO `your-app-service-principal`; GRANT SELECT ON TABLE your_catalog.your_schema.table_2 TO `your-app-service-principal`; **Plus in the UI:** * SQL Warehouses → your warehouse → Permissions → Add service principal → **Can Use** * Genie Space → Share → Add service principal → **Can Run** # Where do you see the real error (not just MessageStatus.FAILED)? The actual error is in the **Genie Monitoring tab** inside your Genie Space — not the App logs. App logs only show the generic `MessageStatus.FAILED`. The Monitoring tab shows the specific SQL error or permission denial that caused it. This single tip would have saved us 3 of those 6 hours. # How long do Databricks permissions take to propagate? Unity Catalog permissions take **5–15 minutes** to propagate. Testing immediately after granting permissions is the most common reason the fix appears not to work. Wait 10 minutes, hard-refresh your browser, then retest. # Quick debug checklist If you're still hitting `MessageStatus.FAILED`: □ Check Genie Monitoring tab for the actual SQL error □ Confirm service principal has USE CATALOG (not just SELECT) □ Confirm SELECT on each table individually — schema-level alone is not enough □ Confirm SQL Warehouse is Running, not Stopped □ Wait 10 minutes after any permission change before retesting □ Hard refresh browser before retesting □ Verify app.yaml uses ["streamlit", "run", "app.py"] — not ["python", "app.py"] # FAQ **Q: Why does Genie work in the UI but fail in the Databricks App?** A: The App runs under a service principal, not your user account. The service principal needs its own separate permission chain regardless of your personal access. **Q: Is USE CATALOG required even if SELECT is granted on all tables?** A: Yes — USE CATALOG is mandatory. Without it the service principal cannot traverse the catalog hierarchy, even with table-level SELECT grants. **Q: Where is the real error when Genie fails in an App?** A: Genie Space → Monitoring tab. App logs only show the generic failure; Monitoring shows the specific cause. **Q: How long do Unity Catalog permissions take to propagate?** A: 5–15 minutes. Always wait before retesting. The Enqurious team wrote up a full walkthrough with SQL commands, screenshots, and the complete [app.py](http://app.py) template Happy to share our Genie Space config or [app.py](http://app.py) in the comments — just ask. **Questions for the community:** * Has anyone else faced similar Genie + App permission issues? * Is there a better way to debug these permission chains? * Would love to hear if Databricks is working on si […truncated]

Azure Databricks Excel Add-in fails in TableSelector even with correct Unity Catalog permissions

https://preview.redd.it/fe1yz91o4f2h1.png?width=800&format=png&auto=webp&s=fb4d699ae3cf49538b46033cc63e2b8aaf44fd95 Hi everyone, I’m having an issue with the Azure Databricks Excel Add-in when trying to query a Unity Catalog table/view from Excel. The user already has the following permissions: * `USE CATALOG` on the catalog * `USE SCHEMA` on the schema * `SELECT` on the table/view * `BROWSE` on the catalog * `CAN USE` permission on the SQL Warehouse However, when using the visual table selector / preview option in the Excel Add-in, Excel shows the following error: Excel Operation Failed Excel operation failed in TableSelector { "ok": false, "status": 200, "statusText": "", "body": { "statement": "SELECT `SOC_ID` FROM `qa_gold`.`bix_cmr_ab_nacional`.`vtx_cmr_gestion_comercial` LIMIT 1000", "warehouseId": "73771902e1b3c9e5" } } The SQL statement shown in the error is: SELECT `SOC_ID` FROM `qa_gold`.`bix_cmr_ab_nacional`.`vtx_cmr_gestion_comercial` LIMIT 1000; I already confirmed that the SQL Warehouse permission is assigned, and the Unity Catalog privileges seem to be correct. Has anyone seen this issue before with the Azure Databricks Excel Add-in? I’m trying to understand whether this is: 1. A permissions issue still missing somewhere, 2. A problem with the Excel Add-in TableSelector / Preview feature, 3. A limitation or bug in the current Excel Add-in, 4. Or something related to the specific view/materialized view being queried. Any suggestions on what else to validate would be appreciated. Thanks!

TruProxy - Live Databricks Cost Estimator - SQL Warehouses

Hi everyone, I'm making progress building TruProxy, a **live cost estimator** for Databricks to get immediate cost estimates **every second** instead of having to wait for the system tables to update. see [Live Cost Estimator : r/databricks](https://www.reddit.com/r/databricks/comments/1t67zhe/live_cost_estimator/) & [TruProxy - Live Databricks Cost Estimator - Clusters : r/databricks](https://www.reddit.com/r/databricks/comments/1tbs0tf/truproxy_live_databricks_cost_estimator_clusters/) I've finished the view for SQL Warehouses and have shared the progress on [YouTube](https://www.youtube.com/watch?v=rkWoj-Z0MyI). We'll continue to share the progress there. I'm looking for **feedback & beta users** and now have a first version to share: [truproxy — live cost visibility for Databricks](https://databricks.trupositive.ai/). Let me know what you think!

Beta alert: Materialized Views and Streaming Tables in Serverless Notebooks

Hi folks, Wanted to share a new feature that's in [beta](https://docs.databricks.com/aws/en/ldp/dbsql/compute#serverless-general-compute) \- creating and refreshing materialized views and streaming tables from serverless compute! Users can create MVs natively in SQL or using `spark.sql("CREATE MATERIALIZED VIEW test_mv AS SELECT * from samples.wanderbricks.booking_updates")` in their notebooks and jobs attached to serverless compute. Workspace admins can enable the beta feature, "MV and ST in Serverless Notebooks and Jobs" in their preview settings. It’s currently available in [select regions](https://docs.databricks.com/aws/en/resources/feature-region-support#serverless-aws). Would love to hear y'all's feedback!

Introducing 5XL SQL Warehouses: A Practical Guide to Meeting SLAs for Your Most Demanding Workloads

Modular structure for Databricks Apps (Streamlit)

Hey, I wanted to share something that's been bugging me for a while and get your take. The official Databricks Streamlit tutorial puts everything into a single **app.py.** Fine for a demo. But the moment a real internal app grows past \~500–600 lines, it stops being fun: * Two people on the team touch the same file → merge conflicts every PR. * Hard to write unit tests when UI, data access, and business logic live in one module. * Git diffs become unreadable, and code review suffers. * When I point Cursor/Claude at the repo, it has to re-read the whole monolith on every prompt. Context window and cost both balloon. So I refactored our internal template into something more boring and modular: app. py # entry point only, routing pages/ ├── home. py ├── analytics. py └── settings. py components/ # reusable UI bits services/ # SQL warehouse / UC / SDK calls assets/ ├── styles.css └── logo.png tests/ *This is my own repo, not a product. Sharing because the single-file pattern bit us hard, and I figured others might find it useful -* [*https://github.com/protmaks/databricks\_apps\_streamlit\_mod\_template*](https://github.com/protmaks/databricks_apps_streamlit_mod_template)

Serverless SQL Warehouses Strategy

Hi, we're a big industrial company and have some pretty diverse use cases in terms of data volume, speed requirements etc. Many of them are quite sporadic (serving data to PowerBI dashboards which are queried a few times per day, but need to be performant then). We are currently thinking on how to provision SQL Serverless Warehouses to our users. How do you do this in your companies: \- Do you have one (or a few) larger warehouses that serve all different use cases? Or \- Do you create / have users create their own warehouses per use case? \- Or do you use a/multiple shared classic warehouses running 24/7? Cost allocation wise the latter one is easier to track, but from a compute cost point of view I imagine the former one is probably more efficient?

Why You Cannot Choose the SQL Warehouse in Databricks Chat & Assistant Features?

How Switching from JDBC/ODBC Clusters to Serverless SQL Warehouses Boosted Our Power BI Performance

pivot() workarounds in Lakeflow Spark Declarative Pipelines

Problem: In Lakeflow Spark Declarative Pipelines, the `pivot()` function is not supported. The `pivot` operation in Spark requires the eager loading of input data to compute the output schema. This capability is not supported in pipelines. Source: [https://docs.databricks.com/aws/en/ldp/limitations](https://docs.databricks.com/aws/en/ldp/limitations) # How can this be mitigated? **Workaround 1: Rewrite PIVOT Using CASE WHEN** This is the most common workaround. You manually expand the pivot into conditional aggregations. >Original Query: SELECT * FROM sales_data PIVOT ( SUM(sales) FOR region IN ('North', 'South', 'East', 'West') ) >Rewritten without PIVOT: SELECT product, SUM(CASE WHEN region = 'North' THEN sales ELSE 0 END) AS North, SUM(CASE WHEN region = 'South' THEN sales ELSE 0 END) AS South, SUM(CASE WHEN region = 'East' THEN sales ELSE 0 END) AS East, SUM(CASE WHEN region = 'West' THEN sales ELSE 0 END) AS West FROM sales_data GROUP BY product This works perfectly in Lakeflow Pipelines because the output schema is fully deterministic at parse time, no eager data loading required. **Workaround 2: Rewrite PIVOT Using aggregate FILTER** Databricks SQL supports the `FILTER(WHERE ...)` clause on aggregates, which is a cleaner alternative to CASE WHEN: >Original PIVOT query: SELECT year, region, q1, q2, q3, q4 FROM sales PIVOT ( SUM(sales) AS sales FOR quarter IN (1 AS q1, 2 AS q2, 3 AS q3, 4 AS q4) ) >Rewritten with FILTER: SELECT year, region, SUM(sales) FILTER(WHERE quarter = 1) AS q1, SUM(sales) FILTER(WHERE quarter = 2) AS q2, SUM(sales) FILTER(WHERE quarter = 3) AS q3, SUM(sales) FILTER(WHERE quarter = 4) AS q4 FROM sales GROUP BY year, region This syntax is often more readable than nested CASE WHEN, especially with multiple aggregations. **Multi-Column PIVOT Rewrite** >For pivoting on multiple columns simultaneously: SELECT * FROM sales PIVOT ( SUM(sales) AS sales FOR (quarter, region) IN ((1, 'east') AS q1_east, (1, 'west') AS q1_west, (2, 'east') AS q2_east, (2, 'west') AS q2_west) ) >Rewritten: SELECT year, SUM(sales) FILTER(WHERE quarter = 1 AND region = 'east') AS q1_east, SUM(sales) FILTER(WHERE quarter = 1 AND region = 'west') AS q1_west, SUM(sales) FILTER(WHERE quarter = 2 AND region = 'east') AS q2_east, SUM(sales) FILTER(WHERE quarter = 2 AND region = 'west') AS q2_west FROM sales GROUP BY year **Multiple Aggregations** You can also rewrite PIVOTs that use multiple aggregate functions. >Original Query SELECT * FROM (SELECT year, quarter, sales FROM sales) AS s PIVOT ( SUM(sales) AS total, AVG(sales) AS avg FOR quarter IN (1 AS q1, 2 AS q2, 3 AS q3, 4 AS q4) ) >Rewritten: SELECT year, SUM(sales) FILTER(WHERE quarter = 1) AS q1_total, AVG(sales) FILTER(WHERE quarter = 1) AS q1_avg, SUM(sales) FILTER(WHERE quarter = 2) AS q2_total, AVG(sales) FILTER(WHERE quarter = 2) AS q2_avg, SUM(sales) FILTER(WHERE quarter = 3) AS q3_total, AVG(sales) FILTER(WHERE quarter = 3) AS q3_avg, SUM(sales) FILTER(WHERE quarter = 4) AS q4_total, AVG(sales) FILTER(WHERE quarter = 4) AS q4_avg FROM sales GROUP BY year **Summary** Both approaches produce identical results and work fully within SDP pipelines with complete lineage tracking.

Live Cost Estimator

I'm building a **live cost estimator** that doesn't have to wait for the system tables or billing data to update. It gives me immediate cost feedback every second and I'm sharing the development journey on YouTube. I already have live costs estimates for **all-purpose clusters, SQL warehouses and interactive serverless compute.** I would love some feedback, suggestions and if you want to try it out or contribute let me know!

Why does the same Databricks SQL query take different time to run?

Pricing for Genie Code: Cluster usage vs. LLM tokens?

Hi everyone, I’m looking into implementing **Databricks Genie Code Agent** in our workspace and I have a question regarding the billing model. My company currently keeps a cluster (SQL Warehouse) running throughout the day. When using Genie Code to ask questions or generate logic, how exactly is the cost calculated? * **Is it just the compute cost?** Since our cluster is already active, does Genie simply "consume" those existing resources to run the generated queries? * **Are there extra LLM costs?** Does Databricks charge a separate fee for the LLM tokens (input/output) used to process natural language, or is the model usage included in the platform fee? Basically, I want to know if using Genie heavily will result in a surprise bill for "AI Tokens" or if it stays within the standard DBU consumption of our active warehouses. Thanks in advance!

Marimo on Databricks

My workflow for a long time involved me switching back/forth between vscode and browser/databricks ui. I like to write my "production code" in normal python, but notebooks are great for exploration, spikes, visualization, triage etc. I could write a small dissertation but for various reasons I don't really like jupyter, and databricks notebooks have their own problems with commented magic commands etc. This led me to check out [marimo](https://marimo.io/), and wow, these are so cool. Code that runs in normal python, merges cleanly, has visualizations, widgets, the the app runs locally and doesn't glitch out, and even the vscode extension works nicely. The problem was, the databricks support wasn't great. It just felt a bit dated. It required a warehouse for sql, doesn't seem to really support serverless, and there were just so many oppurtunities to plug databricks into Marimo. This led me to create [marimo-databricks-connect](https://github.com/brookpatten/marimo-databricks-connect) [pypi](https://pypi.org/project/marimo-databricks-connect/) I tried to plug in "all the things" databricks into the place where they go in Marimo. I'm pretty happy with the result. - Connect to databricks using databricks-connect & spark (not sql warehouse) - Authenticate/configure spark using the default databricks-connect process (env vars, .databrickscfg etc), no additional auth config. - Execution of both python & sql cells - Autocomplete Catalog/Schema/Table/Column Names - Browsing of catalogs/schemas/tables/columns in the marimo data sources view - Browsing of external locations, volumes, dbfs, workspace in the marimo storage browser Notebook widgets to monitor and control of specific instances of databricks capabilities (clusters, workflows, vector search, apps etc) - Widgets to browse & explore databricks capabilities (compute, workflows, unity catalog) - Works in local marimo marimo edit notebook.py, in the vscode extension - Deploy as a databricks app to provide an alternative web based marimo UI. I'm working on adding serving endpoints as AI providers to the notebooks too. In particular what I like to use this for is creating "command center" notebooks for given processes that can include some normal pyspark/sql code to query/triage, widgets to monitor/control various databricks resources, visualizations to monitor dq etc. I just wanted to share and see what the community thinks, would you use it? contributions are welcome. throwaway account because i'm doxing myself via gh repo.

[Passed] Databricks DEA Exam today

https://preview.redd.it/z6mcmrgvmjyg1.png?width=474&format=png&auto=webp&s=28e010f62635d49af3a815998011125d8f2cfa0f Just walked out of the exam and I’m glad to say I passed. I was sweating a bit because the exam content changes on the 4th, so I really didn't want to fail and have to deal with a new syllabus. I've had Databricks at work since late 2023. I’ve been using it because, well, it’s there, but I was mostly just "vibe coding"—picking up some Python and Spark here and there without any real depth. I ran jobs using whatever cluster settings the company gave me without actually knowing what they meant. If you’ve never touched Databricks, this exam is going to be a pain. Even if you’re good at coding, the internal components and the way everything fits together are hard to grasp just by reading. You really need to get your hands dirty in the workspace to get a "feel" for it. **Study Routine** I started with the Databricks Academy stuff, but since I’m juggling work and a toddler, I could only study on weekends. This was a disaster because by the next Saturday, I’d already forgotten what I learned the week before. One month before the exam, I ditched the theory and just hammered Mock Exams. * Udemy is your friend: I bought practice exams from Derar and Santosh. * I snagged them at discounted price. Just wait for the sale if you are not in a hurry. Personally, Santosh’s exams felt closer to the real thing. I saw maybe 5-6 questions that were almost word-for-word. Derar is also solid; honestly, just solve as many problems as possible. Since my study time was limited, I focused on reviewing the questions I got wrong. I realized pretty early that Productionizing Data Pipelines was my weak spot. I didn't try to become an expert in it. I just aimed for a 60% "pass" in that section and doubled down on the areas I was actually good at. Don't completely ignore your weak areas though. If you bomb one section too hard, a couple of silly mistakes in other sections will kill your score. **What's on the exam** The questions are mostly scenario-based. You have to read the prompts carefully. Some things I remember: * Autoloader: This came up a lot. * DLT (now called Lakeflow Spark Declarative Pipelines): should understand what it actually does * Unity Catalog: Permissions (Granting minimum access) and the actual SQL code for it. * Delta Sharing: Knowing the difference between sharing with Databricks vs. non-Databricks users. * Egress Costs: How to avoid them in cross-cloud sharing (Cloudflare R2 was the answer for one). * SQL Warehouses: Classic vs. Pro vs. Serverless. Know when to use which. * DABs (Databricks Asset Bundles): I got at least 3 questions on this. Don't skip it. * Medallion Architecture: It’s not just "what is Bronze/Silver/Gold." They’ll give you a scenario and ask which layer the data should go to next. Also, those "select two" questions are the absolute worst, super confusing. I know the syllabus is changing on the 4th, so I’m not sure how much of this will still apply. But honestly, if you have some background and get familiar with the core concepts, it’s a very doable exam. I’ve learned a lot through this process. Good luck to everyone preparing!

New Databricks Apps: What About Cost at Scale?

I’ve been looking into the new Databricks Apps compute model, and I have one concern.From what I understand, each Databricks App now runs with its own dedicated app compute, rather than simply relying on a shared SQL Warehouse as the main execution layer. I’m wondering what this means at scale. If an organization has dozens or even hundreds of small internal apps, could this become significantly more expensive if each app requires its own compute instead of how it was before all of them sharing a single SQL serverless cluster that can scale to 0? I’d be interested to hear how others are approaching this: Are you consolidating multiple use cases into fewer apps, stopping unused apps, or using another pattern to control costs?

Tried the Lovable + Databricks connector on a hackathon project

I originally thought the Lovable/Databricks connector was kind of a gimmick. Then I had a hackathon project where all the heavy lifting was in Databricks (data processing, enrichment, a bit of ML), but the result had to be shown as a simple app for non-technical users. Tried Lovable mostly out of curiosity, and honestly, it worked better than I expected for an MVP. A couple of practical notes in case anyone else tests it: * service principal needs access not just to the data, but also to the SQL warehouse / compute * I got it working fine on Databricks Free Edition * if you don’t cache responses, repeated queries can get expensive fast because you’re paying for warehouse runtime I still wouldn’t treat this as my default production setup, but for demos / internal prototypes/idea validation, it was surprisingly useful. I wrote a short article with examples - [https://medium.com/@protmaks/databricks-lovable-a-practical-case-study-and-what-it-costs-to-build-an-app-085f61b07126](https://medium.com/@protmaks/databricks-lovable-a-practical-case-study-and-what-it-costs-to-build-an-app-085f61b07126)

Getting started with multi table transactions in Databricks SQL

Transactions let you coordinate operations across multiple SQL statements and tables. All changes succeed together or roll back together, ensuring data consistency across your operations and tables