Vector Search

The Terraform deployment engine is now deprecated; migrate to the direct deployment engine for continued support. AI runtime tasks now automatically package local directories into tarballs during deployment, and the direct engine fixes several convergence issues with empty grants, webhook notification ordering, pipeline configuration, and vector search index creation.

SSH connect now supports specifying serverless usage policies via --usage-policy-id, and bundle deploy/destroy are more robust against transient app states and full workspaces. Bundle validation is stricter for grants while empty field values are now dropped, fixing deployment failures and spurious permission drift.

Scaling document classification to 100k+ labels

Databricks practitioners can scale document classification to over 100,000 labels by pairing SQL-native vector search with the AI Classify function to retrieve and select from a shortlist of candidate labels. Across three benchmarks, this hybrid approach beat the best cost-efficient frontier model by five points of accuracy at roughly a hundredth of the token cost.

Building a soccer coaching app on Databricks

Coach's Corner is a Databricks App that processes 25 fps match tracking data into a sub-second 2D/3D tactical bench with replays, event analytics, a scout chat, and an opponent-dossier agent. The end-to-end solution is powered entirely on the Databricks platform, utilizing Lakeflow pipelines for data refinement, DBSQL and Lakebase for rapid querying, and Unity Catalog-governed AI tools like Genie, Vector Search, and MLflow tracing.

End-to-End RAG Workflow: How Retrieval Augmented Generation Works

Databricks now offers a five-stage RAG workflow for connecting LLMs to external knowledge bases, enabling accurate, domain-specific answers without model retraining. Production RAG requires careful selection of embedding models, vector database indexing, chunking strategies, and hybrid search, with independent evaluation of retrieval precision and generation faithfulness.

What is Vector Search?

Vector search retrieves information based on meaning and context using embeddings, solving keyword-only search limitations for use cases like RAG and recommendations. Databricks AI Search adds reranking, metadata filtering, and automated index updates to improve relevance and simplify operations for production systems.

The CLI improves SSH error reporting with server logs, extends timeouts to 45 minutes for GPU accelerators, fixes environment variable handling to prevent default profile fallback, and corrects Cursor integration hanging. Bundle deployments prevent WAL corruption after failed deploys, fix resource references used multiple times within the same field, add postgres_roles and postgres_databases resources for Lakebase branches, and eliminate spurious recreates when names are backend-normalized.

This release adds new Postgres data API methods to the workspace client, introduces Dynamics 365 as a catalog connection type, and adds capacity reservation and sync configuration fields across compute, database, and vector search services. Additionally, the resource ID field in bundle deployment operations is no longer required, which represents a breaking change.

This release fixes an issue where tag policy and assignment operations failed for hierarchical tag keys containing slashes. It also introduces a breaking change by making the `ResourceId` field optional in `bundledeployments.Operation` and adds support for Dynamics365 connection types and `EndpointId` in vector search indexes.

The direct deployment engine is now generally available and the default for new bundle deployments, while existing deployments continue using their current engine. The CLI adds a quickstart command, version checking, authentication verification fixes for account console profiles, and a new genie_spaces bundle resource for managing Databricks Genie spaces.

This release introduces the AI Search and Bundle Deployments services to the workspace client, while removing the legacy bundle service and package. Additionally, it adds new configuration fields across multiple services, including custom retention hours for catalogs and schemas, auto-clustering options for pipelines, and facet filtering for vector search queries.

The Databricks Python SDK v0.116.0 adds new workspace services for AI Search and Bundle Deployments, along with configuration options for Vector Search facets, Delta Live Tables auto-clustering, and custom catalog retention hours. This release also introduces a breaking change by removing the legacy bundle workspace service and package.

Bring Databricks into Kiro IDE with the AI Dev Kit Power

The Databricks AI Dev Kit Power now offers a one-click setup to integrate Kiro IDE with the full Databricks platform, providing AI-assisted development grounded in your workspace's Unity Catalog metadata. This new path, alongside a lighter PAT-based option, ensures your AI assistant writes SQL with actual columns and respects all row, column, and tag-based grants.

The provider adds service principal Git credential management via principal_id on databricks_git_credential and enables permission management for Agent Bricks resources. Key fixes include metastore external_access_enabled now properly sent in PATCH requests, vector search index timeout increased to 75 minutes and made configurable, and workspace_id now accepting connection IDs alongside numeric workspace IDs.

* The error reported when a direct-only resource (catalogs, external locations, vector search endpoints) is used with the terraform engine now also suggests setting `bundle.engine: direct` in `databricks.yml`, in addition to the `DATABRICKS_BUNDLE_ENGINE` environment variable ([#5295](https://github.com/databricks/cli/pull/5295)).

This release introduces a breaking change that renames the min_qps configuration and CLI flag to target_qps for vector search endpoints, and it stops applying bundle name prefixes to these endpoints. Additionally, it improves the interactive authentication profile pickers, enhances OS keyring token storage behavior, and fixes bundle bugs related to nested notebook directory generation and multi-profile authentication propagation.

This release fixes an issue where the Databricks CLI `--profile` fallback was broken. It also introduces new API methods for `workspaceClient.supervisorAgents()` and `workspaceClient.vectorSearchEndpoints()`, along with several breaking changes related to `Example` and `Tool` fields, and `minQps` in Vector Search endpoints.

This release introduces new methods for managing supervisor agents and vector search endpoints, along with several new fields for connector options, ingestion sources, and customer-managed keys. Breaking changes include the removal of `min_qps` fields from vector search endpoint configurations and making `guidelines` and `description` fields optional in certain services.

The CLI now supports a --limit flag for paginated list commands and caches host metadata lookups for faster repeated invocations. Bundles gain support for Vector Search Endpoints and prompt before destroying Lakebase resources.

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

High-level summary: problems, approaches, and takeways for better RAG with MLflow