Vector Search

End-to-End RAG Workflow: How Retrieval Augmented Generation Works

Databricks now offers a five-stage RAG workflow for connecting LLMs to external knowledge bases, enabling accurate, domain-specific answers without model retraining. Production RAG requires careful selection of embedding models, vector database indexing, chunking strategies, and hybrid search, with independent evaluation of retrieval precision and generation faithfulness.

What is Vector Search?

Vector search retrieves information based on meaning and context using embeddings, solving keyword-only search limitations for use cases like RAG and recommendations. Databricks AI Search adds reranking, metadata filtering, and automated index updates to improve relevance and simplify operations for production systems.

This release adds new methods for managing PostgreSQL data APIs at the workspace level. It also introduces `serverless_compute_id` fields for various Delta Live Tables pipeline operations and `endpoint_id` for Vector Search indexes.

SSH connection error messages are improved with server logs, and the SSH server startup timeout for GPU accelerators is increased. Authentication fallback to default profiles is fixed, and bundle variable references now support Unicode letters.

The Databricks SDK for Java now supports CRUD operations for Postgres data APIs and includes new fields for Azure compute attributes, synced table specifications, and vector search indexes. A breaking change makes the `resourceId` field optional for bundle deployment operations.

The direct deployment engine is now Generally Available and the default for new Databricks deployments, with options to opt out or migrate existing deployments. New CLI commands include `databricks quickstart` for an introduction and `databricks version --check` to report available updates.

This release adds new services for AI Search and Bundle Deployments, along with numerous new fields across various existing services like Catalog, ML, and Vector Search. It also includes a breaking change by removing the `bundle` package and its associated service.

This release introduces new services for AI Search and Bundle Deployments, along with numerous new fields across existing services like Catalog, ML, and Vector Search. The `bundle` package and its associated workspace service have been removed.

Bring Databricks into Kiro IDE with the AI Dev Kit Power

The Databricks AI Dev Kit Power now offers a one-click setup to integrate Kiro IDE with the full Databricks platform, providing AI-assisted development grounded in your workspace's Unity Catalog metadata. This new path, alongside a lighter PAT-based option, ensures your AI assistant writes SQL with actual columns and respects all row, column, and tag-based grants.

You can now manage Git credentials for service principals and permissions for Agent Bricks resources. Key fixes include proper updates for metastore external access, reliable destruction of UC objects, and an increased timeout for vector search index creation.

Bundle users now receive a suggestion to set `bundle.engine: direct` in `databricks.yml` when using direct-only resources with the Terraform engine. Vector search indexes are now supported as a bundle resource (direct engine only), including UC grants and destructive operation confirmations.

This release fixes an issue where the Databricks CLI `--profile` fallback was broken. It also introduces new API methods for `workspaceClient.supervisorAgents()` and `workspaceClient.vectorSearchEndpoints()`, along with several breaking changes related to `Example` and `Tool` fields, and `minQps` in Vector Search endpoints.

This release introduces new methods for managing supervisor agents and vector search endpoints, along with several new fields for connector options, ingestion sources, and customer-managed keys. Breaking changes include the removal of `min_qps` fields from vector search endpoint configurations and making `guidelines` and `description` fields optional in certain services.

The CLI now supports a --limit flag for paginated list commands and caches host metadata lookups for faster repeated invocations. Bundles gain support for Vector Search Endpoints and prompt before destroying Lakebase resources.

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

High-level summary: problems, approaches, and takeways for better RAG with MLflow