Serverless
Recent items mentioning Serverless across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Azure Databricks Serverless is hitting real connectivity friction: users report failures connecting to Azure SQL MI via failover group FQDN 1 and calling SQL Server stored procedures through pymssql 2, while others are stuck simply getting serverless notebook compute to provision 5. On the platform side, both the Go and Java Databricks SDKs shipped v0.162.0/v0.133.0 in July adding serverless compute ID support for job clusters plus effective entitlements data for workspace assignments 67.
Generated daily from the 9 most recent items mentioning Serverless. Click any [N] to jump to the source.
Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via
Multiple IAM list methods now require pagination handling (listDirectGroupMembers, listWorkspaceAssignments, and proxy variants), and the bundleDeployments.createOperation() method was removed. The SDK adds LinkedIn Ads and Marketo connector support plus new fields for Genie conversations and serverless compute IDs.
How to call a SQL Server stored procedure using pymssql from Databricks Serverless Compute?
I have a Databricks notebook that currently uses pyodbc to connect to SQL Server and execute stored procedures. I need to migrate this notebook to Databricks Serverless Compute , so I am looking for an alternative to pyodbc . I am considering using pymssql instead. What is the correct way to connect to SQL Server and execute a stored procedure using pymssql from a Databricks Serverless Compute environment? As of now this is the code we're using: def exec_stored_procedure(stored_procedure, json_data): try: conn = pyodbc.connect(connection_string) cursor = conn.cursor() cursor.execute(f"OPEN SYMMETRIC KEY {Symmetric_name} DECRYPTION BY PASSWORD = '{Symmetric_key}'") cursor.execute("{CALL " + stored_procedure + "}", json_data) conn.commit() except pyodbc.Error as e: print("PyODBC error:", e) except Exception as e: print('An error occured: ', e) finally: try: cursor.execute(f"CLOSE SYMMETRIC KEY {Symmetric_name}") except pyodbc.Error as e: pass except Exception as e: print("An error occurred while closing symmetric key:", e) try: cursor.close() except pyodbc.Error as e: pass except Exception as e: print("An error occurred while closing cursor:", e) try: conn.close() except pyodbc.Error as e: print("PyODBC error while closing connection:", e) except Exception as e: print("An error occurred while closing connection:", e) What would the equivalent implementation using pymssql look like, and are there any additional requirements or limitations when using pymssql with Databricks Serverless Compute? My goal is to replace pyodbc while keeping the existing SQL Server stored procedure logic unchanged.
TutorialsDatabricks All Purpose vs Serverless Computes #databricks #databrickstutorial
Databricks all-purpose compute provides traditional persistent clusters with manual configuration and continuous uptime costs. Serverless compute offers fully managed infrastructure that boots in seconds, scales dynamically, and bills only for active execution time.
How is Serverless Compute implemented under the hood?
Cannot create serverless compute for notebooks
The Databricks Go SDK now exposes effective entitlements data for workspace assignments and adds support for serverless compute IDs in job cluster configurations. These additions enable better management of workspace access controls and serverless compute resources through the SDK.
This release adds effective entitlements to workspace assignment details and serverless compute ID support for job clusters. It also updates model serving telemetry configurations to include fields for table names and telemetry profile IDs.
LakeBridge in Databricks
I am doing data warehouse migration and at the last stage i.e reconcilliation . Now after running this command databricks labs lakebridge configure-reconcile --profile abhi it prompts me for selecting data source,report type,source catalog,target catalog , details for configuring reconcile metadata, and all got installed too but at the end got this error 14:38:46 ERROR [d.l.lakebridge.configure-reconcile] InvalidParameterValue: Only serverless compute is supported in the workspace. So my question is can we not do reconcilliation even after reconcilliation job got failed, if yes then how .
😑Datafactory insists on Cluster compute BUT Databricks defaults to Serverless compute!
Navigating a Synapse Migration to Databricks
Databricks now offers a field-tested playbook for migrating from Azure Synapse (Dedicated SQL Pools, Serverless SQL, and Spark Pools) to a unified Databricks Lakehouse. This phased program helps Synapse customers simplify architecture, improve performance, and lower costs by moving away from a fragmented warehouse not built for modern data workloads.
Tutorials135: Declarative Automation Bundles (Formerly Databricks Asset Bundles)| Part 2 | Sample Project
NewsUnity Catalog Fine-Grained Access Controls on External Engines
Unity Catalog enables fine-grained access controls (FGAC) defined once to be enforced consistently across Databricks and external engines like Apache Spark. External engines can also create and write to UC-managed tables, benefiting from centralized governance, automatic optimization, and transactional safety.
The SDK now provides Data API methods for PostgreSQL in Databricks workspaces. Other additions include capacity reservation support for Azure compute, serverless compute options for pipelines, endpoint ID fields for vector indexes, and Dynamics 365 as a connection type.
This release adds the serverlessComputeId field to Delta Live Tables pipeline configurations, including pipeline creation, editing, cloning, and specification models. Databricks practitioners can now programmatically configure and manage serverless compute resources for their pipelines using the Java SDK.
Announcing the Databricks storage ecosystem: Governing the enterprise data estate, wherever it lives
The Databricks Storage Ecosystem now natively connects hybrid and on-premises storage platforms to Databricks via OpenSharing, enabling centralized data governance and GenAI scaling across your entire hybrid infrastructure. Run Databricks Serverless Compute, Genie, and LLMs directly on your on-premises datasets with a zero-copy architecture, instantly turning isolated data into active, AI-ready assets.
TutorialsSecure Serverless: Azure Private Link Service Direct Connect
The video demonstrates how to set up Azure Private Link Service Direct Connect to enable secure, private connectivity from Databricks serverless compute to any private IP address, such as an on-premises database. It details the architecture, prerequisites, and a step-by-step demo of configuring the Private Link Service and a Databricks Network Connectivity Configuration (NCC) to connect to a MySQL instance.
Rethinking Distributed Systems for Serverless Performance and Reliability
Databricks' serverless compute required rethinking distributed systems to eliminate user-managed infrastructure and improve stability. Architectural innovations like separating applications from compute and intelligent workload routing deliver more stable, predictable, and cost-efficient performance.
Unity Catalog AI 0.4.0
DatabricksFunctionClient now accepts an optional warehouse_id parameter to execute functions via the Statement Execution API when serverless compute is unavailable, while serverless remains the default execution path. The release requires Python 3.10 or later and fixes issues with Gemini toolkit, LangGraph integration, OSS client function creation, and dependency declarations.
NewsDatabricks News: unit testing, OneLake federation, scoped access tokens
Databricks now allows creating Unity Catalog domains for business users, running JAR tasks on serverless compute, and federating OneLake data directly into Databricks. The platform also introduces in-workspace Python unit testing, new data connectors like HubSpot and TikTok Ads, and scoped personal access tokens for enhanced security.
NewsSimplifying Data Pipelines With Lakeflow Declarative Pipelines: A Beginner’s Guide
NewsServerless as the New "Easy Button": How HP Inc. Used Serverless to Turbocharge Their Data Pipeline
Unity Catalog AI 0.3.1
Unity Catalog AI 0.3.1 automatically initializes DatabricksFunctionClient in Databricks environments, returns function results as native Python types instead of strings, and improves SQL NULL parameter handling. The release also enhances serverless endpoint connection recovery with better retry logic and optimizes Spark session creation to initialize on-demand.
Assessment workflows require account groups to be created beforehand, and Service Principal is no longer supported for workspace installations—account-level installations now require Service Principal with Account Admin and Workspace Admin privileges. Table migration with default catalogs is fixed, workflow assessment filters to the last 30 days, migration progress workflows pause by default, and account group conflicts produce warnings instead of errors.
Unity Catalog AI 0.3.0
Functions in Unity Catalog AI 0.3.0 now execute in a sandbox by default rather than the main process, with local and legacy execution modes available. The release adds callable and source retrieval APIs for functions, fixes serverless credential issues, and streamlines toolkit initialization for Databricks compute.
News
