Databricks SQL
Recent items mentioning Databricks SQL across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
What is Databricks SQL?
Databricks SQL is the data warehouse side of the platform. It runs directly on your data lake and supports ANSI SQL with Delta Lake extensions, so you get warehouse-style SQL over the tables you already have instead of loading copies into a separate system. Queries run on SQL warehouses, compute dedicated to SQL workloads.
It carries the tooling analysts expect: a SQL editor with AI assistance and version history, AI/BI dashboards, alerts that monitor query results and deliver notifications, and a metrics layer for defining business metrics with consistent calculations. It also covers a fair amount of ETL, since streaming tables and materialized views can be defined and refreshed directly in Databricks SQL and queries can be scheduled as jobs.
SQL warehouses come in serverless, pro, and classic types, and Databricks recommends serverless when available: capacity management, patching, upgrades, and performance optimization are handled by Databricks. Query history, execution plan inspection, and automatic recommendations for inefficient queries are built in.
What's the difference between serverless, pro, and classic SQL warehouses?
All three run Databricks SQL queries. Databricks recommends serverless SQL warehouses when available, citing instant and elastic compute, minimal management overhead, and lower total cost of ownership, since capacity management, patching, upgrades, and performance optimization are handled by Databricks. Pro and classic remain as alternatives where serverless isn't offered.
Can I use Databricks SQL for ETL, not just BI queries?
Yes. You can define and refresh streaming tables and materialized views directly in Databricks SQL, and schedule SQL queries as jobs for automated data processing and reporting workflows.
Do I need to move my data into Databricks SQL first?
No. Databricks SQL is built on lakehouse architecture and runs directly on your data lake with ANSI SQL and Delta Lake extensions, so the warehouse sits on data where it already lives rather than on copies loaded into a separate system.
Does Databricks SQL include dashboards and alerts?
Yes. AI/BI dashboards with AI-assisted authoring are built in, and alerts can monitor query results, evaluate conditions, and deliver notifications automatically. There's also a metrics layer for defining business metrics with consistent calculations.
Sources: Databricks SQL overview (Databricks docs) · SQL warehouses (Databricks docs)
Serverless Databricks SQL analytics is now available in the Azure Databricks on Azure Government beta, bringing commercial platform parity and U.S.-only data residency 4. On the infrastructure tooling side, version 1.144.0 of the Databricks Terraform provider expanded statement_timeout support across all SQL warehouse types and ensures the configuration is preserved during resource exports 2.
Generated daily from the 10 most recent items mentioning Databricks SQL. Click any [N] to jump to the source.
We wrote about this
- Brickster SQL: learn SQL by building a city under the seaA simple guide to Brickster SQL, our free iPhone app for learning SQL ten minutes a day: what it teaches, how lessons work and where to get it.6 min read
- Databricks Vector Search is now AI Search, and the rename is not cosmeticFull-text indexes with no embeddings, two SQL functions that do different jobs, and a scaling switch that landed separately. What changed on 1 June 2026 and what you now have to decide.7 min read
The databricks_app resource now includes a thumbnail_path argument to upload local image files as application thumbnails. Support for statement_timeout has been expanded across all SQL warehouse types and is now preserved during resource exports.
The new Azure Databricks on Azure Government is now in Beta
Azure Databricks on Azure Government is now in Beta, bringing commercial platform parity with Unity Catalog governance, serverless Databricks SQL analytics, and U.S.-only data residency. Upcoming releases on the path to General Availability will add AI capabilities like Genie, Agent Bricks, and Model Serving, with FedRAMP Class D and IL5 authorization expected by early 2027.
Seeking feedback on my Databricks data engineering portfolio project
Hi everyone, I’m preparing to apply for junior/entry-level Data Engineer roles focused on Databricks, and I’d appreciate honest feedback on a portfolio project I’ve been building. It is a governed lakehouse project for a synthetic table-tennis e-commerce dataset: - Deterministic input contract and synthetic fixture with intentional duplicates, malformed records and quality issues. - Bronze ingestion designed around Databricks Asset Bundles, Auto Loader, append-only raw tables, lineage metadata and retained checkpoints. - Silver transformation with explicit validation rules and a quarantine layer. - Gold publication with privacy controls: only aggregated, non-PII cells that meet a k-anonymity threshold are published. - Unity Catalog / Terraform / dedicated service principal / least-privilege model. - Manual serverless Databricks Job with materialization followed by a read-only verifier. - 287 local tests covering contracts, transformations, bundle configuration, verifier logic and Terraform privilege rules. The current DEMO-001 cloud milestone produced: - Silver: 15 orders, 10 customers, 7 products - Quarantine: 6 expected rejected rows - Gold: €640 revenue, 5 orders, 0 refunded lines - Pre-suppression controls: €812.20, 13 orders, 2 refunded lines - Status: COMPLETED_WITH_QUARANTINE I have deliberately kept the scope honest: this is a fixed deterministic snapshot, not CDC, streaming, SCD, generic incremental processing, SQL Warehouse, Power BI, or a production monitoring setup. Repository: tt-ecommerce I would especially value feedback on: Does this demonstrate relevant Databricks/Data Engineering capability? What is missing before using it in applications? Would you prioritise finishing governance/consumer access, adding SQL Warehouse/BI, or CI/CD and operational monitoring? Is the documentation clear, or does it feel over-engineered for a portfolio project? What would you expect to see in a strong junior Databricks project that is not here yet? Thanks — direct criticism is welcome. submitted by /u/Alone-Cow-1563 [link] [comments]
Jobs now check the data before they run (SQL conditions). How would you use it?
We shipped SQL conditions for Lakeflow Job triggers to beta a little while ago, and we want to hear how people would use it plus share a few patterns that work. Docs here: https://docs.databricks.com/aws/en/jobs/trigger-sql-conditions The idea. A trigger tells the job when to run. A SQL condition tells it whether to run. You attach a saved query and a SQL warehouse to a trigger (cron, table, file etc.) and when the trigger fires the query the condition first. https://preview.redd.it/x11wrj7g7u0i1.png?width=1012&format=png&auto=webp&s=9ec41109617eef506116bfc2515bd0d8c525b5f6 There are two modes: Query returns any rows : No rows, no run. Result value changes : the job runs when the return value differs from the last check. Here are a few examples: Freshness plus volume. Today's partition exists and has enough rows, This returns zero rows until the data is ready SELECT 1 FROM main.sales.orders WHERE order_date = current_date() HAVING count(*) > 100 Quality gate. Run only when today's rows have no nulls in a key column. So if there are no bad rows, the job runs SELECT 1 FROM main.sales.orders WHERE order_date = current_date() HAVING count(*) > 0 AND count_if(customer_id IS NULL) = 0 Fan-in. Five upstream feeds write a status row when they finish. Run once all five are done. This replaces the "sentinel table plus a cron that polls it" setup a lot of people run today. Note: if you are watching job completions, then just use job completion triggers! SELECT 1 FROM ops.batch_status WHERE batch_date = current_date() HAVING count_if(status = 'complete') = 5 Watermark advanced. Use "Result value changes" with a one-row, one-column query. The job runs whenever the vendor delivers a new batch. SELECT max(batch_id) FROM main.bronze.vendor_feed Things to know. Each check requires a warehouse and will show up in query history. A check times out at 5 minutes including warehouse startup. Checks run as the job's run-as identity. What is coming next : See why a run did not start. We want each evaluation to show when the query ran, the row count or value it returned, why it did or did not start a run and any error. Parameters. Job parameters inside the condition query, with per-trigger overrides. This is for everyone promoting the same job through dev, staging and prod with bundles, where catalog and schema names change per environment and nobody hand-edits prod. A test button that actually runs the query and shows you the result and whether the job would start right now. Today it only checks that the query exists and has no parameters. Read the condition from a .sql file in the workspace instead of only a saved query, same as SQL tasks already do. A default warehouse so you do not have to pick one every time. Questions for you: What does "ready" mean for your most annoying upstream? Row count, a flag table, a max timestamp, something else? Which of the upcoming items would you use first, and what did I miss? Try it out on Genie code too! Give genie a prompt and see if it picks up your intent as a sql condition. submitted by /u/saad-the-engineer [link] [comments]
ADF Lookup Activities Against Databricks Are Using Shared Compute. Looking for Better Alternatives. Post
Hi everyone, I'm working on an Azure Data Factory (ADF) ingestion framework where we currently use ADF Lookup activities to retrieve metadata used for pipeline orchestration. The challenge is that the lookups are querying Databricks-managed metadata and are executing through Shared Compute, which has become a performance bottleneck when running multiple pipelines concurrently. The requirement we're evaluating is to replace the Lookup activities with Databricks notebooks/jobs running on Serverless compute to improve concurrency and reduce dependency on Shared Compute. My concern is around consuming the results back in ADF. Today, the Lookup activities return a structured output that is consumed directly by downstream activities such as ForEach loops and dynamic expressions. The proposed notebook approach would: Accept input parameters. Execute the metadata query. Build a JSON payload that matches the Lookup activity output format. Return the payload using dbutils.notebook.exit(). However, while ADF can consume output directly from a Databricks Notebook activity, my understanding is that Databricks Job activities mainly return execution metadata rather than the notebook payload itself. I also looked into whether a Databricks Serverless SQL Warehouse could be used for the metadata queries instead of replacing the Lookup activities entirely, but it appears the Databricks Delta Lake connector may not support SQL Warehouses in the same way it supports clusters and compute resources. So I'm trying to understand the best architecture for: Improving lookup performance. Leveraging Databricks Serverless compute. Preserving the existing downstream behavior. Avoiding major ADF pipeline refactoring. Has anyone implemented a similar pattern? Specifically: Have you successfully replaced ADF Lookup activities with Databricks Jobs while preserving Lookup-style outputs? What is the best way to return structured metadata from Databricks back into ADF? Would you recommend: Databricks Notebook Activity Databricks Job Activity Persisting results to storage and reading them back Another pattern entirely? Any recommendations or lessons learned would be greatly appreciated. submitted by /u/Extreme-Title6444 [link] [comments]
Databricks SQL: Check Join Row Counts Before Resizing the Warehouse
I'm building an AI data engineer that runs as a Databricks App. Here's what broke along the way
I'm the founder of AI Data Engineer, so take this with that in mind. I've worked with data for over ten years, first at a bank, then as a team lead. One thing never changed: the business asks for one new table, and it waits weeks for an engineer. So I built a Databricks App for that. You describe the table in plain English, the agent studies your Unity Catalog tables and asks business questions, and after you approve, it builds a Data Vault and the final mart as serverless jobs and keeps them updated on schedule. A few lessons from the last weeks: The model should not write SQL. It only proposes a plan: entities, keys, joins, rules. Plain code turns that plan into hubs, links, history satellites and the mart. Same plan, same tables, every time. Samples lie. My first checks ran on samples. On a 7.5M row table they said only 27% of orders matched a customer. The real number was 100%. Now every check runs on the full table through the SQL warehouse. The model doesn't need your data. It sees table structure and statistics, never values, and it's called through a Model Serving endpoint in your own workspace. If a rule needs a value, like "which status means paid", it asks a human. Open models aren't ready for this yet. I replayed real planning requests on the Qwen and GPT-OSS endpoints in Databricks. Valid JSON, but they skipped the business questions and guessed. One grouped monthly revenue by month without year, so March 1997 and March 1998 would be one row. Change Data Feed deletes hide the worst bugs. Delete keys came back as 40.0000000000 while the hub had 40, and 2,398 rows were marked deleted. Since then every build is reconciled against the source. Where it is now: a 30M row TPC-H mart matched a direct query on the source with zero mismatches across 3,213 checked cells, and an incremental build after 1,000 updates and 100 deletes matched exactly. Once a mart is built, one click makes an AI/BI dashboard and a Genie space on top of it. submitted by /u/No_Advertising2536 [link] [comments]
Use Serverless SQL Warehouses to bypass workspace cluster startup latency
For transparency, I work at Abilytics, a data and AI consulting firm. Serverless SQL Warehouses keep compute instances warm and ready, eliminating the cold-start delay of traditional JDBC/ODBC or classic clusters. When building BI dashboards or running ad-hoc queries, standard warehouses often waste minutes spinning up nodes. Serverless decouples the control plane and compute plane, provisioning resources instantly. Configure your warehouse type to 'Serverless' in the UI or via the Databricks SQL API using 'warehouse_type': 'SERVERLESS'. This ensures concurrent queries scale out instantly without waiting for node provisioning, dropping your p95 latency significantly for bursty analytical workloads. submitted by /u/AbilyticsEng [link] [comments]
Agentic code converter is GA for SQL
Migration · GA Agentic code converter is GA for SQL Genie Code converts T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata SQL to Databricks SQL. It validates each file and fixes errors before writing to a workspace folder. Work in a migration project, or run /migrate in a Genie Code chat for one file or folder. Afterward, keep chatting to clear action items. The Notebooks target is not part of this GA. https://preview.redd.it/u5m0quj71fyh1.png?width=1808&format=png&auto=webp&s=1a5f16a1d66fa88e9bc171e10efb16c33dadfa2b Looking to migrate workloads but don't have the labor to do so. This is going to continue to expand its SQL dialect but taking an agentic approach to migrations really speeds up migrations and cuts down on time/money from labor submitted by /u/Wise_Ear_4064 [link] [comments]
How to debug remote_query() performance?
I have a job which queries some Azure SQL DBs connected through UC managed connections (no lakehouse federation) via remote_query() function, and facing pretty long execution times (several minutes) when reading a few thousand rows only. When running the same query via SSMS on the target host, it returns same result within milliseconds. Is there any way how I can debug the pure execution time from Databricks side? I'd like to understand if the queries run slow, or if there's any Databricks or network overhead accounting responsible. Note that remote_query() runs T-SQL in this scenario, not Databricks SQL. So there's no SQL conversion involved at any point in time. submitted by /u/Sea_Basil_6501 [link] [comments]
Show HN: 10x faster, 3x cheaper ETL
For the past few months our team has been working on an agentic framework that takes large, expensive SQL workloads and compiles them to Rust, backed by a formal proof of correctness. This is the sort of work I used to do by hand as a systems performance engineer, but now it happens in hours instead of months. We first redid our internal pipelines with this technique and they're now 10x faster and 3x cheaper. Next, we went live with our Snowflake and Databricks SQL customers. They're seeing similar results with critical production workloads that originally cost 6-7 figures per year. We'll be publishing benchmarks and code samples over the coming weeks; for now, you can read a bit more (and watch our anime launch trailer) on the site. Happy to answer questions about how it all works. --- top comments --- [gtirloni] Why was SQL slow? Where did processing shift to?
A full month-end finance app built on Databricks Apps
Below are the things covered in the video: 1. Overview: drag the month slider and every figure on the page moves with it, including a cash flow diagram from sales through to profit 2. P&L: the statement drawn as a ladder from revenue down to profit, plus a bridge showing why we beat or missed budget 3. Cafes: ranking, a map of the Front Range, a margin heatmap by month, and how fast each new cafe paid back its fit-out 4. Menu: which items earn their place and which are just popular 5. Cash: a 13-week forecast, overdue invoices by age, and the supplier payment calendar 6. Close: the month-end checklist as a board, with how long each close took historically 7. What-if: move price, volume, wage and ingredient levers and watch profit change 8. A one-page monthly report that prints to A4 Under the hood: seven Delta tables in Unity Catalog, read through a SQL warehouse, with the app running as a Databricks App in Python. All the finance logic (consolidation, budget variances, the close tracker, the cash forecast) sits in one file that a finance analyst could read. submitted by /u/oneaffidavit1 [link] [comments]
Release v0.169.0 adds sdpSource to the ML DataSource, and adds customerVersionLabel, maxDbu, and reydenClusterSizeMultiplier to SQL warehouse create, edit, and get responses. It also removes the replicationConfig field from CreateShare and ShareInfo, which is a breaking change for Sharing API users.
Adds SdpSource to ml.DataSource and CustomerVersionLabel, MaxDbu, and ReydenClusterSizeMultiplier to SQL warehouse requests and responses. Breaking: removes ReplicationConfig from sharing.CreateShare and sharing.ShareInfo.
Release v0.155.0 adds customer_version_label, max_dbu, and reyden_cluster_size_multiplier to the SQL warehouse create, edit, and get calls, plus an sdp_source field on the ML DataSource. It also has breaking changes: the replication_config field is removed from the sharing CreateShare and ShareInfo types.
The Go SDK introduces methods for ML experiment issue tracking and sandbox file operations, along with support for Realtime SQL warehouses and job duration targets. Breaking changes remove the EffectiveServerlessComputeId field from job tasks and replace ReadMask with View across Agentkit memory requests.
Databricks Genie One — How to Restrict Agent Creation and Scheduled Tasks?
We use Databricks Genie One, and some users are creating unnecessary Genie Agents and scheduled tasks. Removing Consumer access or Databricks SQL access affect users’ ability to view dashboards, run SQL queries, or use other Databricks features Questions: Can workspace admin prevent regular users from creating Genie Agents? Can scheduled-task creation be disabled while users can still chat with Genie? submitted by /u/--playground-- [link] [comments]
This release adds support for real-time SQL warehouses, issue management methods for experiments, and replication configuration for Delta Sharing. It also introduces a breaking change by removing the effectiveServerlessComputeId field from job run tasks.
What happens when too many ppl run queries on the same Databricks SQL Warehouse?
Do the queries just wait in a queue, or does the warehouse automatically handle the extra load.. Also, how do you figure out when it's time to increase the warehouse size instead of letting queries wait... Curious how this works in real projects. submitted by /u/Bhanuprakash_1947 [link] [comments]
How to set budgets and alerts on SQL warehouses so self-service never surprises Finance
The SDK adds account-level management for identity visibility filters and introduces statement timeout configuration for SQL warehouses. It also adds parent path support for pipelines, custom template formats for SQL alerts, and streaming shuffle partition settings for machine learning features.
The SDK adds the AgentKit service, account-level identity visibility filter APIs, and statement timeout options for SQL warehouses. It also introduces parent path support for pipelines and removes the Mason service in a breaking change.
This release adds the Agent Kit workspace service, account IAM identity visibility filters, and statement timeout configuration for SQL warehouses. It also introduces a breaking change by removing the Mason workspace service and package.
Set Budgets and Alerts for Cloud Data Warehouse Costs
Eliminate runaway data warehouse costs by tagging SQL warehouses at creation and establishing layered budgets with pacing alerts that project month-end overruns days in advance. Monitoring system.billing.usage through custom dashboards allows organizations to bring unattributed spend to zero and turn warehouse costs into a metric every team actively owns.
Auto stop option on databricks apps
Want to request a feature for databricks app, it should have an auto stop option. I know Databricks markets it as a web app that should be readily available, but I think customers deserve the option to turn auto-stop on or off, like we do for, say, SQL Warehouses. This has been a major blocker for us suggesting apps to leadership, the engineering team has build some really useful apps for usecases like Data modelling and stuff which is used when needed but they running 24*7 all time is wasting compute and getting flagged in finops P.S i know we can have a databricks job which can shut down and start the app during working hours but we have a global team working round the clock so it's not feasible for us. If we have someone from databricks apps team this is a highly requested thing from alot of people around the community. submitted by /u/sumit671 [link] [comments]
Rebuilding an 800-Line PySpark Pipeline in 200 Lines of Databricks SQL
system.query.history is now Generally Available in Databricks
Until now, answering "which query is eating our warehouse?" or "who changed that table?" meant clicking through the Query History UI one workspace at a time. Now it is a SELECT. The table logs every statement run on SQL warehouses, serverless compute, and Lakeflow pipelines, across all workspaces in the same region. Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-dataengineering-systemtables-ugcPost-7511094425977016320-L7uj/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]
Laya off the benchmark: can a zero-shot decision model route real SQL traffic?
Weekend-ish experiment on Databricks. One table, TPC-H orders, 15M rows, living in two places at once: Lakebase (Databricks' managed Postgres), a continuously synced copy with a btree index on the key. Sub-100ms point lookups, useless for a GROUP BY over 15M rows. Delta behind a serverless SQL Warehouse. Great at scans and aggregations, slow at fetching one row. The synced table is the nice part: native continuous Delta to Lakebase Postgres (needs a PK + CDF), so it's one dataset under one Unity Catalog, replication handled by the platform instead of a homemade pipeline. Then I put laya (convaiinnovations/laya, a non-autoregressive zero-shot decision model, vanilla, no fine-tuning) behind a FastAPI Databricks App. It reads each query and picks the engine. MLflow traces input to decision to execution. I submitted by /u/Limp-Park7849 [link] [comments]
New Databricks SQL window fuction: counter_diff
Top 5 System Table Queries for Understanding Your Databricks Costs
Five targeted system table queries deliver near-real-time visibility into your Databricks spend broken down by product, SQL warehouse, and team. Deploy them to flag cost spikes against a rolling 14-day baseline with DBSQL alerts and project 30-day expenditures in a single query using AI_FORECAST.
What if Databricks showed the cost directly next to every job and run?
Databricks give us a lot of cost data, but usually not where we actually need it. When I open a job run, I can see duration, compute, tasks and logs, but not what that run actually costs. When I create a cluster or SQL warehouse, I choose the size first and DBU, and discover the real cost later. That’s the problem we’re trying to solve, bringing cost directly into the Databricks UI. We created a free browser extension, with no backend or telemetry https://finopsway.com/ submitted by /u/Significant-Guest-14 [link] [comments]
Before you move a job to serverless: the limitation list is the real decision, not the startup time
Most classic-vs-serverless comparisons stop at "serverless starts faster and Databricks manages it". True, but that is not what decides it in practice. Where they differ structurally: with classic compute the control plane asks your cloud provider for VMs, and the cluster runs in your own account and network, so you pick instance types and own the networking, and you pay the cloud provider for machines plus Databricks for DBUs. With serverless, the docs say resources run in "the serverless compute plane, which is managed by Databricks" and you provision nothing in your own cloud account. Your tables stay in your own storage in both cases. What actually decides it is the limitation list: - R is not supported - compute-scoped libraries and compute-scoped init scripts are not supported - only Spark Connect APIs are available, so the Spark RDD APIs are out - streaming: Trigger.AvailableNow() is supported, Trigger.Continuous(interval) and Trigger.ProcessingTime(interval) are not - maximum runtime is 7 days; a run that exceeds it is terminated and not retried So: serverless for SQL warehouses, ad hoc exploration and spiky jobs where waiting for a cluster is the real cost. Classic when you need a specific instance type or GPUs, cluster-scoped libraries, R, RDD code, a continuous stream, or your own network setup. I drew both paths side by side as a short animated diagram (2:36): https://youtu.be/s8452E2Y6WM Series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4 For those who have moved production jobs across: what caught you out that is not on the documented list? submitted by /u/Broad-Cherry-5999 [link] [comments]
Databricks pipeline works in one workspace but fails in another — where would you start?
Imagine the same pipeline and code are deployed to two Databricks workspaces. Workspace A: catalog.schema.table → Query works → Pipeline works Workspace B: catalog.schema.table → Catalog not accessible → Pipeline fails The catalog exists, the table exists, and the SQL itself is valid. Would you first investigate: Workspace-to-catalog assignment Unity Catalog permissions USE CATALOG USE SCHEMA SELECT privileges Service principal permissions External locations Storage credentials Cluster access mode SQL Warehouse identity I'm curious about the actual troubleshooting order people follow for these kinds of Unity Catalog issues. What would you check first? submitted by /u/hanshu6576 [link] [comments]
Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?
Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ai_query ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]
How do you parse an xml that's in string format?
I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this: .... ... ... Regex is not an option. Substring is not an option. Because it's so messed up it's not consistent. I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql Please, help you mate. submitted by /u/pineapple_brownies [link] [comments]
No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring
https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]
Databricks is too expensive for small teams" is usually a workload problem, not a platform problem
Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]
Do small companies actually use Databricks?
Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand. My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses. Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency. When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs. Would love to know what you guys think. submitted by /u/Puzzled-Mail-9092 [link] [comments]
What is a Warehouse in Databricks SQL?
Query works in Databricks SQL but fails through JDBC
I’ve noticed cases where a query runs successfully in the Databricks SQL editor but fails when the exact same query is executed through a JDBC-based application or data quality tool. Has anyone run into this? Was the issue related to query wrapping, session settings, SQL dialect differences, or JDBC driver behavior? How do you usually troubleshoot these cases? submitted by /u/No_Ambition8323 [link] [comments]
Metric views now replace existing tables or views using a backup-and-create strategy rather than direct create-or-replace syntax. Lazily formatted Databricks SQL log records are also now properly interpolated when mirrored into dbt logs.
What type of compute do you guys use in Databricks?
Hey everyone, I've been using Databricks for a while, but I'm still curious about how other teams decide which compute option yo use. There are so many choices now - serverless,job clusters, all-purpose clusters, SQL warehouses, etc. - and sometimes it's not really obvious which one makes the most sense. And when do you go with serverless vs a normal cluster? I'm also wondering if people are mainly choosing based on cost, performance, startup time, or just what their team is already comfortable with. Would be interested to hear what you're using in your projects and what made you choose it. Especially if you've switched from one type of compute to another and noticed a real difference. submitted by /u/Delulu62134 [link] [comments]
Does Photon support CSV?
Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]
Moving ~150 Tableau dashboards to Databricks AI/BI - anyone done this at scale?
We are considering moving from Tableau to Databricks AI/BI Dashboards and wanted to see if anyone here has made a similar move. We currently have around 150 Tableau dashboards used by 1,000+ users, with a mix of technical and non-technical users. Most of the underlying data already lives in Delta tables, so naturally we're looking at whether it makes sense to consolidate and use Databricks for the dashboarding layer as well. For those who have done this, how has the experience been? Anything you wish you knew before migrating? Any major limitations compared to Tableau, especially around visualization, performance, or the experience for non-technical users? I'm also curious about cost at scale. With 1,000+ users potentially hitting dashboards, did SQL warehouse usage become a concern? Would you feel comfortable replacing Tableau completely with AI/BI Dashboards today, or do you still find yourself needing Tableau for certain use cases? Would really appreciate hearing about real-world experiences. submitted by /u/Immediate_Bus5250 [link] [comments]
databricks SQL UDF in select statement
"List columns" statements when querying Databricks via ODBC
I've got an old .NET app running on Windows Server. The app submits queries to a serverless SQL Warehouse. I've noticed that each query ( SELECT statement) submitted by the app results in three statements in the SQL Warehouse. One statement containing the query submitted by the app, and two statements that look like Listing columns 'catalog : foo, schemaPattern : foo, tablePattern : resultssettable, columnName : null' . The query might take 200ms to execute. The "Listing columns" statements take about 90ms each to execute. In aggregate, these "Listing columns" statements double the duration of queries sent to this warehouse. How do I get rid of these freak'n "Listing columns" statements? Here is what I've done/tried: Tables in the query are full qualified with catalog and schema. Query specifies the names of the columns to retrieve (i.e., not a SELECT * FROM ... ). I have Use Native Query enabled in the DSN. I have Fast SQLPrepare enabled in the DSN. Enabling Ignore Tables Metadata from All Schemas in the DSN makes no difference. I configured the Databricks ODBC driver to pool connections. I have the 64-bit Databricks ODBC driver installed, version 2.11.00.1005. I don't have access to the .NET apps source. I have been using the following PowerShell script to recreate the issue and test configuration changes: $dsn = "DSN=Test Connection" $sql = @" SELECT col1 , col2 , ... FROM dev.foo.bar WHERE whatever = 1; "@ $conn = New-Object System.Data.Odbc.OdbcConnection $dsn try { $conn.Open() $cmd = $null $cmd = $conn.CreateCommand() $cmd.CommandText = $Sql $cmd.CommandTimeout = 30 $adapter = New-Object System.Data.Odbc.OdbcDataAdapter $cmd $adapter.MissingSchemaAction = [System.Data.MissingSchemaAction]::AddWithKey $table = New-Object System.Data.DataTable "foobar" [void] $adapter.Fill($table) return $table.Rows.Count } finally { if ($reader) { $reader.Close() } if ($cmd) { $cmd.Dispose() } } I feel like I missing […truncated]
Databricks SQL Alert - Scheduling Questions
Databricks SQL Alert — Question Set Help needed, I created a SQL alert in databricks to notify me when a new entry is added to a dataset/table within the last 24 hours. SQL SELECT COUNT(*) AS new_ FROM catalog.schema.table WHERE xyz IN ('a', 'b', 'c', 'd') AND datetime >= CURRENT_TIMESTAMP() - INTERVAL 24 HOURS; Within the Alerts panel, I set the condition as follows: First row Column name: new_ Operator: > Static value: 0 I then added my email address in the Notifications section and set "When alerting, notify" to Always . I then selected View Alert , where I created a schedule — let's say every day at 9:00 AM . Questions a) How exactly does the schedule work here? I am asking because I selected "Always" under the notification settings, but I have also created a schedule to run every day at 9:00 AM. What is the relationship between these two settings? b) Let's suppose I build this alert and create the schedule as described above. Person B (sales dept rep) also subscribes to this alert. I do not want Person B to receive an alert unless the new record belongs to the Sales department. For example, if a new record is added that belongs to a department other than Sales, and the scheduled alert runs at 9:00 AM, will Person B still receive an email notification? Thanks in advance for any clarification! submitted by /u/SquareLong2523 [link] [comments]
Busting SQL Migration Myths: How New SQL Features Make Lift-and-Shift to Lakehouse Easier
New Databricks SQL Scripting features enable direct lift-and-shift migrations of legacy PL/SQL, allowing teams to translate procedures, cursors, temp tables, and transactions line for line without rewriting business logic. Once migrated, these procedures retain their original control flow while gaining built-in Unity Catalog lineage and access control.
Databricks Dashboard - Scheduling Questions
( FYI, I did write the initial draft of this message by myself, and then later asked AI to proof read it and later pasted the same here, Apologies in advance!) I recently developed and published a Databricks dashboard for a personal project, and I want to learn how the scheduling feature works. For context, I have read the Databricks documentation, Manage scheduled dashboard updates and subscriptions | Databricks on AWS , but I couldn't find answers to the questions below. Databricks Dashboard — Question Set 1 I published the Databricks dashboard using Individual Data Permissions . I then added another person as a participant, created a schedule, and subscribed to the schedule from both accounts. Let's call them Person A (publisher + subscriber) and Person B (subscriber). Questions a) Assume Person B works in the Sales department and should only be able to see Sales data. Since the schedule was created by Person A, when the scheduled run occurs and Person B receives the email notification they subscribed to, will the PDF attachment contain all departments' data , or will it contain only the data Person B is permitted to see? b) After the scheduled run, if Person B opens the dashboard using the link in the email notification, or accesses the dashboard directly through Databricks, what data will they see? Will they see all department data , or only the data they are permitted to see based on their individual data permissions? My understanding is that when Person B visits the dashboard, it does not automatically refresh just because a scheduled dashboard update has occurred. I may be misunderstanding how this works, so I'd appreciate some clarification. Databricks SQL Alert — Question Set 2 I also created a SQL alert to notify me when a new entry is added to a dataset within the last 24 hours. Please note: The screenshot below is not from my actual example. I am only including it so that new developers like myself can relate to what I am referring to. enter image descr […truncated]
Tagging individual ai_query() calls in SQL?
Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps. We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step. From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL. Has anyone found a workaround, or is moving the calls out of SQL currently the only option? Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc submitted by /u/hulioshort [link] [comments]
Load Testing Databricks SQL Warehouses with JMeter — Part 3: Running and Analyzing Results
Load Testing Databricks SQL Warehouses with JMeter — Part 2: Concurrency and the Test Plan
Load Testing Databricks SQL Warehouses with JMeter — Part 1: Inputs and Configuration
Serve Tableau Reports Directly from Databricks SQL
Understanding EXPLAIN FORMATTED in Databricks SQL
Building a soccer coaching app on Databricks
Coach's Corner is a Databricks App that processes 25 fps match tracking data into a sub-second 2D/3D tactical bench with replays, event analytics, a scout chat, and an opponent-dossier agent. The end-to-end solution is powered entirely on the Databricks platform, utilizing Lakeflow pipelines for data refinement, DBSQL and Lakebase for rapid querying, and Unity Catalog-governed AI tools like Genie, Vector Search, and MLflow tracing.
Introduces catalogs.yml v2 support, a skip_optimize config for opting out of post-materialization OPTIMIZE, and Rust kernel backend for SQL warehouses. Fixes numerous incremental model bugs around constraints and tags, but now requires --full-refresh to apply changes to primary/foreign key expressions.
DBSQL MCP output limit
Querying Metric Views via Classic/Pro SQL Warehouses
Incremental REPLACE WHERE Flows Brings Targeted Refreshes to SDP and DBSQL
Get Tuesday's version of this
Tracking Databricks SQL? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.