Serverless

Databricks Serverless Compute: Practical Recommendations, Cost-Performance Guidance, and When to Use Serverless Instead of SQL Serverless Warehouses or Classic Clusters

Databricks Serverless compute is a snappy choice for every-day queries with some cost trade-offs. For ETL workloads, it is both fast + cost-efficient. In either scenario and unlike common misconception, there is no cluster fine-tuning needed. Some more detailed takeaways from some testing on my coffee dataset + practical guidance below. Quick disclaimer: I work for a Databricks aligned consulting company. All reporting and recommendations here in any direction are independently made. General Observations from Performance Testing 1) Speed wise, much to my surprise, Serverless beat some of my previous Databricks’ SQL Serverless performance testing. Not across all queries, but in nearly half. 2) Databricks’ SQL Serverless compute is still the best compute option from a pure cost-perspective when it comes to “every day” analytics needs, but from an ETL perspective, “Serverless” is very comparable, without the trial & error of even doing t-shirt sizing, let alone knobs. 3) The scaling time to support an initial large workload was noticeable on the first query and does have room for improvement. I did a dummy print() statement right before (which took like 1-2 secs to warm up). Then hitting that first query containing the joins and 7B+ rows fact table clearly gave the engine something to chew on. For reference, on my SQL Serverless test a while back, the same query took about 20 secs on a large, 15 on XL. Practical Serverless Recommendations 1) If I was orchestrating ETL workloads with SQL and/or Python, I’d use Serverless compute as the cost of Serverless for jobs is pretty darn affordable. My concerns around costs are completely negated when it comes to jobs compute. For new ETL workloads, Serverless is an easy choice for me. For existing ETL workloads, I’d try out to see if Serverless lowers your costs and/or improves performance. Also explore any potential compatibility issues. If you have heavily fine-tuned workloads, that don’t really require much maintenance, migrating the workload might not be best. 2) For daily Python work on established Databricks environments, I would consider trying Serverless. Not having to wait for clusters to boot up and fine-tuning is likely going to lead to productivity gains offsetting any higher costs from interactive Serverless compute itself. No more “left the cluster on by accident” or “I oversized it” type of situations. That said, there are orgs with very experienced folks where cluster management feels like second nature to them. 3) If I was exclusively working with analytical SQL workloads, I’d still use Databricks’ SQL Serverless compute for my day to day queries + dashboards. 4) For folks new to Databricks, I would generally stick to “Serverless” compute all around, and use the “SQL Serverless” warehouses only for BI type experiences. Closing Thoughts on Serverless Having very performant compute without worrying about any configurations (other than rate limits to give you cost controls if you wish) is quite nice from a productivity standpoint. No node count, DBR versioning, t-shirt sizing, nothing. You have plenty of compute options, with or without fine-tuning ! I stress this "no fine-tuning" because there is a plethora of LLM generated articles outdated on Databricks' compute options. Thank you for reading! *No AI was used to write or assist in writing this post. For better or for worse. **Dataset and queries used, configured for the 7B+ row count: https://github.com/joshbogran/coffeeshopdatageneratorv2 submitted by /u/JosueBogran [link] [comments]

Unable to Start Serverless Compute in Databricks Free Edition

Cost-optimized way to reflect source DB changes in Silver in <1 minute?

Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute. Currently, the flow looks like this: Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode. The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow. This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need. However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that. My question is : is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer? Maybe Lakeflow Connect or some custom process? Extra notes: - I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently. - I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute. - I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale. submitted by /u/CyberEnzo [link] [comments]

Serverless compute resolves some public domains but not others

What’s new in Databricks - August 2026

Databricks shipped many major Generally Available features in August 2026. Here is the breakdown of what just landed: 🚀 Unity AI Gateway Enterprise AI governance layer covering model access, Model Context Protocol (MCP) management, and cost observability. 🔒 Role-Based Access Control (RBAC) Switch to scoped, temporary role assumptions instead of dealing with permission bloat. 🔑 Secrets in Unity Catalog Unified security secrets are now governed, 3-level namespace securable objects. ⚙️ Serverless Compute Access Control Granular admin controls over who can trigger serverless workloads across your organization. ⚡ Lakebase Postgres APIs & LTAP Direct Writes Accelerated synced-table loads and improved transactional data integration. 🤖 Genie Agent Upgrades Official GA releases for both the Agent mode API and Full-page Genie Code view. submitted by /u/Youssef_Mrini [link] [comments]

Clarification on Automating Serverless Compute Permissions via SDK/API

Error While Running Workloads on Databricks Free Edition – Serverless Compute

Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]

&quot;List columns&quot; statements when querying Databricks via ODBC

I've got an old .NET app running on Windows Server. The app submits queries to a serverless SQL Warehouse. I've noticed that each query ( SELECT statement) submitted by the app results in three statements in the SQL Warehouse. One statement containing the query submitted by the app, and two statements that look like Listing columns 'catalog : foo, schemaPattern : foo, tablePattern : resultssettable, columnName : null' . The query might take 200ms to execute. The &quot;Listing columns&quot; statements take about 90ms each to execute. In aggregate, these &quot;Listing columns&quot; statements double the duration of queries sent to this warehouse. How do I get rid of these freak'n &quot;Listing columns&quot; statements? Here is what I've done/tried: Tables in the query are full qualified with catalog and schema. Query specifies the names of the columns to retrieve (i.e., not a SELECT * FROM ... ). I have Use Native Query enabled in the DSN. I have Fast SQLPrepare enabled in the DSN. Enabling Ignore Tables Metadata from All Schemas in the DSN makes no difference. I configured the Databricks ODBC driver to pool connections. I have the 64-bit Databricks ODBC driver installed, version 2.11.00.1005. I don't have access to the .NET apps source. I have been using the following PowerShell script to recreate the issue and test configuration changes: $dsn = &quot;DSN=Test Connection&quot; $sql = @&quot; SELECT col1 , col2 , ... FROM dev.foo.bar WHERE whatever = 1; &quot;@ $conn = New-Object System.Data.Odbc.OdbcConnection $dsn try { $conn.Open() $cmd = $null $cmd = $conn.CreateCommand() $cmd.CommandText = $Sql $cmd.CommandTimeout = 30 $adapter = New-Object System.Data.Odbc.OdbcDataAdapter $cmd $adapter.MissingSchemaAction = [System.Data.MissingSchemaAction]::AddWithKey $table = New-Object System.Data.DataTable &quot;foobar&quot; [void] $adapter.Fill($table) return $table.Rows.Count } finally { if ($reader) { $reader.Close() } if ($cmd) { $cmd.Dispose() } } I feel like I missing […truncated]

Serverless in Compute Section

It is great to see serverless in the compute section. Especially because I proposed it some time ago and lobbied for it during advisory meetings. Now we can set permissions for serverless compute. More options will come. I can say that it is my favorite recent improvement. submitted by /u/hubert-dudek [link] [comments]

Anyone else gotten a rough surprise with Databricks costs once things hit production?

This keeps coming up in conversations with clients and I feel like it's worth its own thread. The pattern is almost always the same. A pipeline gets built to bring in data for analytics or ML, works fine in testing, then goes to production and the compute bill is way higher than expected. Nobody budgeted for it because on paper it looked like a simple ingestion job. Usually the real issue isn't Databricks itself, it's the ingestion design. The repeat offenders I keep seeing: Full reloads instead of proper CDC, so you're paying to process data that hasn't even changed. Serverless SQL running more often than needed, because someone assumed near real time was required when batch every few hours would've worked fine. No plan for schema evolution, so jobs fail or reprocess more than they should every time something shifts upstream. Cluster sizing set for peak load "just in case" instead of actual daily volume. Most of the fix comes down to being honest about the freshness you actually need. A solid CDC layer feeding into something like Kafka before it hits Databricks tends to cut a lot of the unnecessary compute, since you're only moving what changed. Curious what caused it for others, ingestion design or job scheduling? submitted by /u/Only-Dragonfruit4130 [link] [comments]

Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via

How to call a SQL Server stored procedure using pymssql from Databricks Serverless Compute?

I have a Databricks notebook that currently uses pyodbc to connect to SQL Server and execute stored procedures. I need to migrate this notebook to Databricks Serverless Compute , so I am looking for an alternative to pyodbc . I am considering using pymssql instead. What is the correct way to connect to SQL Server and execute a stored procedure using pymssql from a Databricks Serverless Compute environment? As of now this is the code we're using: def exec_stored_procedure(stored_procedure, json_data): try: conn = pyodbc.connect(connection_string) cursor = conn.cursor() cursor.execute(f&quot;OPEN SYMMETRIC KEY {Symmetric_name} DECRYPTION BY PASSWORD = '{Symmetric_key}'&quot;) cursor.execute(&quot;{CALL &quot; + stored_procedure + &quot;}&quot;, json_data) conn.commit() except pyodbc.Error as e: print(&quot;PyODBC error:&quot;, e) except Exception as e: print('An error occured: ', e) finally: try: cursor.execute(f&quot;CLOSE SYMMETRIC KEY {Symmetric_name}&quot;) except pyodbc.Error as e: pass except Exception as e: print(&quot;An error occurred while closing symmetric key:&quot;, e) try: cursor.close() except pyodbc.Error as e: pass except Exception as e: print(&quot;An error occurred while closing cursor:&quot;, e) try: conn.close() except pyodbc.Error as e: print(&quot;PyODBC error while closing connection:&quot;, e) except Exception as e: print(&quot;An error occurred while closing connection:&quot;, e) What would the equivalent implementation using pymssql look like, and are there any additional requirements or limitations when using pymssql with Databricks Serverless Compute? My goal is to replace pyodbc while keeping the existing SQL Server stored procedure logic unchanged.

How is Serverless Compute implemented under the hood?

Cannot create serverless compute for notebooks

LakeBridge in Databricks

I am doing data warehouse migration and at the last stage i.e reconcilliation . Now after running this command databricks labs lakebridge configure-reconcile --profile abhi it prompts me for selecting data source,report type,source catalog,target catalog , details for configuring reconcile metadata, and all got installed too but at the end got this error 14:38:46 ERROR [d.l.lakebridge.configure-reconcile] InvalidParameterValue: Only serverless compute is supported in the workspace. So my question is can we not do reconcilliation even after reconcilliation job got failed, if yes then how .

😑Datafactory insists on Cluster compute BUT Databricks defaults to Serverless compute!

Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data

Serverless compute outbound IP whitelisting for external API calls

Handle case issue in column names

I am loading data using pyspark with spark_reader.load(data_path) However, in some cases data can be very messy, with fields using different case for each rows (can be in nested structs). Here is an example of data : [ { &quot;field_1&quot;: &quot;1&quot;, &quot;Field_2&quot;: 1, &quot;field_3&quot;: &quot;b&quot;, &quot;field_4&quot;: [{&quot;A&quot;: 1, &quot;b&quot;: 2}, {&quot;A&quot;: 3, &quot;b&quot;: 4}], }, { &quot;Field_1&quot;: &quot;2&quot;, &quot;Field_2&quot;: 2, &quot;Field_3&quot;: &quot;BB&quot;, &quot;Field_4&quot;: [{&quot;a&quot;: 1, &quot;B&quot;: 2}, {&quot;a&quot;: 3, &quot;B&quot;: 4}], }, ] In this case, the load fails with following error : pyspark.sql.utils.AnalysisException: Found duplicate column(s) in the data schema: `field_1`, `field_3`, `field_4` And I can't find a clean way to handle this case. I tried the following workaround : raw = spark.read.text(data_path) normalized_rdd = raw.rdd.mapPartitions(_normalize_partition) raw_df = spark.read.json(normalized_rdd) With a python function _normalize_partition that normalizes the column names. However it does not work in my case as I use a Databricks serverless compute and the use of .rdd is not allowed. [NOT_IMPLEMENTED] Using custom code using PySpark RDDs is not allowed on serverless compute.

Struggling to learn Spark UI on Databricks, all tutorials are outdated. Any good resources?

Hey everyone, I'm fairly new to Spark and trying to understand how it actually executes jobs specifically the DAG visualization, stages, task metrics, and executor stats in the Spark UI. The problem I'm running into: almost every video tutorial I find was recorded on an older version of Databricks, and the UI looks completely different from what I see today. The gap is big enough that I can't follow along at all. A few specific issues I've hit: \- \`spark.databricks.io.cache.enabled\` throws a CONFIG\_NOT\_AVAILABLE error on newer runtimes \- \`spark.catalog.clearCache()\` throws a NOT\_SUPPORTED error because I'm on Serverless compute (Community Edition) \- The Spark UI itself looks different from what tutorials show I'm using Databricks Community Edition (free tier), which I've now learned only gives Serverless compute so some things just aren't available. My questions: 1. Is there a good up-to-date resource (video, blog, or docs) for understanding the Spark UI on the current Databricks version? 2. For learning Spark internals (DAG, stages, task metrics), is it better to just use local Spark or Google Colab instead of Databricks free tier? 3. Any tips for following older Spark UI tutorials and mentally mapping them to the current UI? Thanks in advance!

Can't start Serverless Warehouse after extend budget

Serverless Compute connectivity issues with .com.br domains vs. Classic Clusters Spark hangs

Beta alert: Materialized Views and Streaming Tables in Serverless Notebooks

Hi folks, Wanted to share a new feature that's in [beta](https://docs.databricks.com/aws/en/ldp/dbsql/compute#serverless-general-compute) \- creating and refreshing materialized views and streaming tables from serverless compute! Users can create MVs natively in SQL or using `spark.sql("CREATE MATERIALIZED VIEW test_mv AS SELECT * from samples.wanderbricks.booking_updates")` in their notebooks and jobs attached to serverless compute. Workspace admins can enable the beta feature, "MV and ST in Serverless Notebooks and Jobs" in their preview settings. It’s currently available in [select regions](https://docs.databricks.com/aws/en/resources/feature-region-support#serverless-aws). Would love to hear y'all's feedback!

Inquiry regarding Serverless outbound IP ranges and Serverless compute firewall configuration

Serverless SQL Warehouses Strategy

Hi, we're a big industrial company and have some pretty diverse use cases in terms of data volume, speed requirements etc. Many of them are quite sporadic (serving data to PowerBI dashboards which are queried a few times per day, but need to be performant then). We are currently thinking on how to provision SQL Serverless Warehouses to our users. How do you do this in your companies: \- Do you have one (or a few) larger warehouses that serve all different use cases? Or \- Do you create / have users create their own warehouses per use case? \- Or do you use a/multiple shared classic warehouses running 24/7? Cost allocation wise the latter one is easier to track, but from a compute cost point of view I imagine the former one is probably more efficient?

How Switching from JDBC/ODBC Clusters to Serverless SQL Warehouses Boosted Our Power BI Performance

Live Cost Estimator

I'm building a **live cost estimator** that doesn't have to wait for the system tables or billing data to update. It gives me immediate cost feedback every second and I'm sharing the development journey on YouTube. I already have live costs estimates for **all-purpose clusters, SQL warehouses and interactive serverless compute.** I would love some feedback, suggestions and if you want to try it out or contribute let me know!

Serverless compute throws OUT_OF_MEMORY exception

Any tips for DABs in CI/CD? Seems pretty useless so far.

We've used DAB-commands like Validate and Plan for a while - to print Github PR-comments on what the PR will change, delete and create. But we are struggling to catch breaking changes before they are committed to main. Some examples: After migrating a pipeline to serverless compute, our branch passed the plan in the PR stage, but failed in main due to `You must use the Advanced edition when using serverless compute. (400 INVALID_PARAMETER_VALUE)` which is something I would expect CI to catch. Another example is Lakeflow generating a new pipeline-ID, which means during deploy it will try to apply itself to an existing pipeline and fail on mismatching pipeline-ids. Again, would've loved to fail in CI instead of main. How are you solving this?

DABs Python Mutators: Stop Copy-Pasting the Same Config Across 50 Jobs

# Situation You've got 30, 50, maybe 100 jobs in your Declarative Automation Bundle. Every single one needs failure notifications. Every single one needs cost-center tags. Every single one needs the right cluster policy. And every time someone adds a new job, they forget at least one of those things. You could write **one Python function** that enforces it automatically at deploy time. That's what DABs Python mutators do. # What Are Mutators? A mutator is a Python function that runs during `databricks bundle deploy`. It receives every job (or pipeline) in your bundle, whether defined in YAML or Python, and returns a modified copy. Think of it as middleware for your deployment config. Write a tag, permission, or compute standard once, and apply it automatically to every resource at deploy time. No drift. Decorate a function with  \`@job\_mutator\`, \`@pipeline\_mutator\`,  \`@schema\_mutator\`, or \`@volume\_mutator\`. The function receives the resource + bundle context, and returns a transformed copy. You register them in `databricks.yml`: python: mutators: - 'mutators:add_pipeline_mutators' # Example This example defines common pipeline standards for every pipeline in your bundle: * Specifies common tags. * Enforces serverless compute. * Defines default notifications group and when to trigger an alert. &amp;#8203; from databricks.bundles.core import Bundle, pipeline_mutator @pipeline_mutator def add_pipeline_mutators(bundle: Bundle, p: Pipeline) -&gt; Pipeline:     p = replace(p, tags=_add_common_tags(bundle, p.tags))     p = replace(p, serverless=True)     default = Notifications.from_dict(        { "email_recipients": "${var.recipients}",          "alerts": ["on-update-failure", "on-update-fatal-failure", "on-flow-failure"] }     )     p = replace(p, notifications=[default])     return p Other resources: * The [bundle-examples](https://github.com/databricks/bundle-examples) repo has a working example at [knowledge\_base/job\_programmatic\_generation](https://github.com/databricks/bundle-examples/tree/main/knowledge_base/job_programmatic_generation). * And as well the documentation page: [https://docs.databricks.com/aws/en/dev-tools/bundles/python/#modify-resources-defined-in-yaml-or-python](https://docs.databricks.com/aws/en/dev-tools/bundles/python/#modify-resources-defined-in-yaml-or-python) # Use Cases https://preview.redd.it/2v2ikiexd4yg1.png?width=632&amp;format=png&amp;auto=webp&amp;s=fa39246e3830b857f1da43777aacfb1079a261a8 Job Mutator Examples: * Enforce default email notifications, owners, tags. * Standardize job clusters / serverless environments. * Inject common job parameters or health/queue settings. Pipeline Mutator Examples: * Enforce pipeline cluster / environment settings. * Apply consistent configuration, catalog/schema, or triggers across all pipelines. Schema Mutator Examples: * Apply standard permissions or tags to all schemas. * Enforce naming conventions or lifecycle settings. Volume Mutators: * Set default storage locations, ACLs, or lifecycle flags. * Add org‑wide tags or conventions to all volumes.

VNet Data Gateway unable to connect to Azure Databricks Serverless SQL via Private Endpoint

Heading into the May 2026 Databricks Data Engineer Associate Exam? Read this first.

So if you've been scrolling through older study guides for the Databricks Data Engineer Associate exam — be careful. The syllabus got a pretty big update this month, and the focus has shifted toward the platform's newer declarative features. I spent some time going through the new guidelines. Here's what I found. Lakeflow is the new standard. The exam has moved away from manual ETL logic. You need to understand Lakeflow Spark Declarative Pipelines (formerly DLT) and how Streaming Tables and Materialized Views actually differ. If your notes still say "DLT" everywhere, time to update them. DABs are no longer a side topic. Databricks Asset Bundles — basically infrastructure-as-code for workflows — is now a core part of the exam. They want to see that you can deploy through DABs, not just click around the UI. Unity Catalog is the default assumption. No more legacy Hive Metastore questions. The exam lives in a UC-enabled world now. Three-tier namespace (catalog.schema.table), Volumes for unstructured data, column-level lineage — that's where your time should go. Serverless Compute is showing up more. When do you pick Serverless SQL Warehouses or Serverless Jobs over classic clusters? That tradeoff — less config overhead vs. less control — is fair game now. The weightings that surprised me → 31% on Processing (Lakeflow, Spark, Streaming Tables) → 18% on Productionizing (DABs, Workflows, deployment) That's almost half the exam right there. Honestly, if you just understand why Databricks is pushing toward declarative tools — letting the platform handle the boring parts so you can focus on the actual logic — a lot of the questions start to make sense. For practice material, BricksNotes has an updated practice test that follows the May 2026 format — 45 questions, 90 minutes, same weightings. → [bricksnotes.com/blog/databricks-data-engineer-associate-new-exam-guide-may-2026](http://bricksnotes.com/blog/databricks-data-engineer-associate-new-exam-guide-may-2026) Good luck to everyone testing this month! Drop questions below if you're stuck on any of the new topics — happy to help where I can.