Databricks SDK

The Databricks Python SDK introduces a new list_commands method to the workspace sandbox service. This update allows practitioners to programmatically list and inspect commands within Databricks sandbox environments.

The AI functions service now supports an ai_decide method, and multiple Jobs and Tasks models now accept environment variable configurations. Additionally, command_path in DeploymentSpec and api_secret_ref in SchemaRegistryConfig are no longer required fields.

The Databricks SDK for Go now supports Config.Headers to configure custom HTTP headers on all client requests in addition to normal authentication. Practitioners can use the new StaticHeaders helper to easily apply a fixed set of headers.

The Databricks Python SDK introduces skill management operations to the AI Gateway workspace client. Users can now create, retrieve, list, update, finalize, and delete skills programmatically using new methods on the AI Gateway service.

AI Runtime commands have moved from experimental to databricks air, and SSH commands now support keeping detached background processes running after tunnel disconnect. Databricks Asset Bundles direct engine resolved multiple issues around unnecessary resource recreations, Unity Catalog grant convergence, and Git-sourced Python tasks.

MLflow 2.11.5

Users can now opt in to route Unity Catalog model registry artifact uploads and downloads through the Databricks SDK Files API. This update provides an alternative transfer mechanism for managing model artifacts within Databricks environments.

This release adds support for the new Mason workspace-level service via the w.mason client. It also introduces service_credential and secret_reference fields to Unity Catalog model provider configurations.

This release adds AI Gateway credential management APIs for MCP services and expands data pipeline connectors to include TikTok Ads and Smartsheet as ingestion sources. It also introduces Avro and Protobuf transformer format support for pipelines, timezone configuration for ML job scheduling, and new securable types for agent services and skills.

The SDK adds new credential management methods for MCP services in AI Gateway and expands job notifications to include maintenance-related events. Support for additional data formats (Avro, Protobuf) in pipelines and new connection types (TikTok Ads, Smartsheet) extend integration capabilities.

Direct deployment state version 3 is a breaking change requiring Databricks CLI v1.8.0 or later. New capabilities include Docker credential helpers for Databricks Artifact Registry, AI Gateway service support in bundles, file-change job run triggers, and bundle improvements for resource reference handling, deployment reporting, and state management.

Add GroupName field for pipelines.RunAs.

The SDK adds support for new fields across multiple services: AI Runtime jobs can now specify image paths and priority classes, ML models support budget policies and tags during publishing, and serving configurations gain workspace credential options including new GPU accelerator types. Pipelines add development mode configuration and workspace settings gain ingress network policy controls.

SSH sessions now automatically reattach and replay missed bytes when the tunnel connection drops temporarily, preventing session interruptions and eliminating manual reconnect workarounds. The CLI adds PyDABs secrets support, JSON output for aitools install with error categorization for CI automation, and displays bundle sync progress by default.

Release introduces three new workspace services (AI Functions, Domains, Sandbox), extends Feature Engineering with new backfill and operation management methods, and adds AWS Secrets Manager and Azure Key Vault connection support. Multiple breaking changes remove fields from model serving and catalog configurations that may require code updates.

The SDK adds a purge_feature_entities() method for feature engineering and introduces new deny policy type support with a deny field in PolicyInfo. New budget_policy_id and tags fields are available for IngestionConfig and MaterializedFeature to enable budget policy tracking and resource tagging.

The SDK adds a new domains workspace-level service for managing domains. ML Feature Store capabilities are expanded with new fields for feature filtering, scheduling, and improved source column specifications across Delta tables and Kafka sources.

The SDK adds an execute_command_sync() method for sandbox and new fields for ML features (mode, time_window, full_feature_name). Breaking changes remove routing and rate limiting configuration fields from model serving APIs.

The Pipelines API's source_table and source_schema fields are now optional to support streaming and message-bus connectors that don't use source tables or schemas. New ai_functions and sandbox workspace services are available, along with expanded APIs for bundle deployments, feature engineering, PostgreSQL, IAM v2, and model deployment resources.

This release adds support for Postgres snapshot operations, RabbitMQ pipeline ingestion, external IAM entity lookups, and H100 GPU compute. It also introduces breaking changes by removing several ML service fields and making source schemas and tables optional in pipeline specifications.

Bundles automatically migrate from Terraform state to the direct engine on clean deploys (opt-out via engine: terraform), and SSH sessions stay connected during idle periods with automatic keepalives every 30 seconds. Direct engine bundle operations now correctly handle removed configuration fields, support cluster policies resources, and include a DATABRICKS_BUNDLE_RESOURCE_MAX_WAIT timeout setting.

Databricks Asset Bundles now report detailed resource action summaries and file sync counts during deploy and destroy operations, alongside fixes for workspace Git folder metadata and state migration. The CLI now enables FIPS 140-3 compliance by default for TLS connections, adds Goose support to aitools, and automatically resolves conflicting databricks-connect pins in local environment setups.

Add EnabledTelemetryFeatures field for serving.TelemetryConfig.

Fixes OAuth token cache conflicts in multi-process deployments and Spark runtime version handling, with breaking changes requiring code updates for pagination in IAM list operations. Introduces git-based deployment for apps and custom UDF support in ML functions.

Go version requirement raised to 1.25, and IAM list methods now include pagination with breaking signature changes, while CreateOperation is removed from BundleDeployments. Fixes concurrent OAuth token cache handling and Spark runtime selection for major-only versions.

The SDK makes action_type and status fields optional instead of required in bundle deployment operations. The create_operation() method is removed from the bundle_deployments workspace service.

The SDK adds LinkedIn Ads and Marketo connector options plus PostgreSQL halfvec and varchar type support for synced tables. AccountIamV2API.list_workspace_assignments and WorkspaceIamV2API.list_workspace_assignments_proxy now require pagination handling, introducing breaking changes that may affect existing code.

The SDK adds pagination to four IAM methods (list_direct_group_members and list_workspace_assignment_details in both account and workspace APIs), which is a breaking change requiring code updates. New fields are added for Genie conversation analysis, group membership tracking, serverless compute identification, and Postgres pipelines.

The SDK adds new methods for managing groups, users, service principals, and workspace assignments at both account and workspace levels, providing expanded IAM capabilities. The NewCluster field in JobCluster is now optional, a breaking change that affects existing cluster configuration code.

The SDK adds comprehensive IAM v2 methods for managing users, groups, service principals, and workspace assignments at both account and workspace levels. The new_cluster field in JobCluster is no longer required, which is a breaking change.

Bump github.com/databricks/databricks-sdk-go from v0.166.0 to v0.170.0 (#6251).

The CLI adds databricks environments setup-local to provision matched Python environments for Databricks compute targets and extends aitools install to support Gemini CLI and Pi. Bundles fix the ignored bundle.deployment.lock.force setting, add pipeline cascade_on_destroy control, improve experimental job_runs with idempotency tokens and completion waiting, and add UC secrets resource support.

The SDK adds job triggers for scheduling, API source connectors for pipelines, secret value retrieval in requests, new GpuXlarge8 workload types, Netsuite connection support, and token forwarding for apps. Breaking changes remove browse-related fields from model and MCP service APIs and make several user and service principal fields required.

The Terraform deployment engine is now deprecated; migrate to the direct deployment engine for continued support. AI runtime tasks now automatically package local directories into tarballs during deployment, and the direct engine fixes several convergence issues with empty grants, webhook notification ordering, pipeline configuration, and vector search index creation.

The Databricks SDK adds principal-level overrides and resource-type tracking to billing configurations, plus a new block_usage billing action type. New features include NetSuite connection support in the catalog, grant-level policy control, and input/output column mapping for pipeline transformers.

The Go SDK adds fields for granular billing control: PrincipalOverrides, ScopeType, and ResourceType for alerts and budgets, plus a new BlockUsage action type for alert configuration. The catalog API now supports grant management for policies via a new Grant field and PolicyTypeGrant enum type.

Breaking changes include removal of direct AWS and Azure credential fields from model provider configurations and State field type changes in bundle deployment operations. New features add KinesisStreamConfig support for ML streaming sources, a Mode field for pipeline updates, and Pi coding agent attribution in the User-Agent header.

The SDK adds new fields to job triggers (PauseStatus), workspace objects (DirectoryInfo), and IAM entities (GroupId, ServicePrincipalId, UserId) for enhanced API access. A breaking change removes the InternalId field from IAM Group, ServicePrincipal, and User types, requiring code updates for any existing consumers.

The SDK introduces a new AI Gateway workspace service and adds new methods and fields across bundle deployments, compute, jobs, serving endpoints, and other APIs. Breaking changes include a modified create_deployment() method signature and removed deployment_id and lifetime fields.

The Go SDK adds AwsAccessKey authentication for Amazon Bedrock model providers and EntraServicePrincipal authentication for Azure OpenAI and Microsoft Foundry providers. These new configuration options enable additional credential management methods when integrating external AI models through Databricks.

SSH connect now supports specifying serverless usage policies via --usage-policy-id, and bundle deploy/destroy are more robust against transient app states and full workspaces. Bundle validation is stricter for grants while empty field values are now dropped, fixing deployment failures and spurious permission drift.

The release introduces a new workspace-level AI Gateway service for managing AI catalog operations. Bundle Deployments now support update operations and include additional fields for tracking version history, dashboard metadata, and modification details.

The Go SDK adds a PatchTelemetryConfig method for managing serving endpoint telemetry configuration. New fields expand pipeline schema specs with FanoutOptions and SQL alerts with Parameters support.

New special destination enum values enable more granular catalog ownership assignments at the schema, table, volume, function, and model levels. Breaking changes to BundleDeployments require updating your deployment creation calls due to argument order changes and removal of the DeploymentId field.

The SDK adds DependencyMode field support across cluster management APIs along with new fields for AI runtime tasks, schema registry configuration, and synced table management. A breaking change removes the Lifetime field from ml.TimeWindow that will require code updates.

The databricks genie ask command is now stable (promoted from experimental), enabling practitioners to ask natural-language questions about their data directly from the CLI. Multiple reliability and usability improvements ship, including auth profile validation timeouts, fixes for GCS-backed UC Volumes deletion, bundle validation for sql_warehouse configurations, opt-in spark_python_task file downloads in bundle generation, and instance_pools resource support in Declarative Automation Bundles.

The Databricks Go SDK now exposes effective entitlements data for workspace assignments and adds support for serverless compute IDs in job cluster configurations. These additions enable better management of workspace access controls and serverless compute resources through the SDK.

The SDK adds new fields for job performance targets, ML stream and window configuration, VPC endpoint information, and compliance standards across multiple services. Breaking changes affect IAM v2 API method paths and ML window configuration, requiring code updates to remove WindowDuration as a required field and handle the removal of LongRolling support.

The SDK adds support for new clean room task run management and PostgreSQL CDF configuration, plus numerous new fields across compute, ML, pipelines, and serving services. Breaking changes include IAM v2 method endpoint updates, type changes for internal IDs, and removal of deprecated fields in jobs and ML services.

The SDK adds workspace-level grants listing methods and new schema fields for jobs, ML experiments, serving endpoints, and workspace configuration. Breaking changes include removal of fields from secrets and ML services, and making the postgres role field required.

This release adds SDK support for JAR analysis assets, review workflows, and task run listing in Databricks Clean Rooms, alongside a new JDBC connection type for catalog integrations. Additional updates introduce Azure capacity reservation group settings for compute instance pools, initial parent paths for bundle deployments, and lifetime attributes for ML time windows.

This release introduces a breaking change by removing the CodeSourcePath field from AI runtime job tasks. It also adds the EffectiveWorkspaceId field for disaster recovery stable URLs and the SourceMetadataColumn field for Delta Live Tables pipeline configurations.

The Clean Rooms API now supports shared output settings, notebook asset metadata, and collaborator aliases. Pipeline configurations for Google Ads, Meta Marketing, and TikTok Ads now include custom report options.

Add CreateCdfConfig, DeleteCdfConfig, GetCdfConfig, GetCdfStatus, ListCdfConfigs and ListCdfStatuses methods for w.Postgres workspace-level service.

Introduces catalogs.yml v2 support, a skip_optimize config for opting out of post-materialization OPTIMIZE, and Rust kernel backend for SQL warehouses. Fixes numerous incremental model bugs around constraints and tags, but now requires --full-refresh to apply changes to primary/foreign key expressions.

Profile-based authentication now takes precedence over environment variables, and deployment bugs affecting PyDABs-generated permissions and resource grants with --select have been fixed. New experimental job_runs resources are available, along with UC volume path support and options to hard-delete Lakebase branches or assume control of existing postgres databases on bundle deploy.

The Databricks Go SDK updated job run structures to include deployment and version ID fields. MLflow experiment objects now support trace locations, and disaster recovery stable URLs include a stable workspace ID field.

This release adds an UpdateTime field to bundle deployment resources. As a breaking change, the Role field is now required in PostgreSQL database specifications.

The SDK now supports specifying a parent path for jobs and associating a Git credential ID with workspace repositories. Model serving configurations now include CpuLarge and CpuMedium options for workload types.