Scaling document classification to 100k+ labels
Summary
Databricks practitioners can scale document classification to over 100,000 labels by pairing SQL-native vector search with the AI Classify function to retrieve and select from a shortlist of candidate labels. Across three benchmarks, this hybrid approach beat the best cost-efficient frontier model by five points of accuracy at roughly a hundredth of the token cost.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Improving Lakebase Postgres compute cache
Lakebase Postgres now runs an autoscaling cache that works alongside shared buffers to keep more data resident on compute instead of falling back to the storage layer. In production this delivered 2x throughput, fewer storage-layer reads, and lower latency compared to standard Postgres caching.
Five AI Questions We're Hearing from Financial Services Leaders
Transitioning financial services AI from pilot projects to governed production deployments hinges on addressing five key questions shaping the industry. Learn how Databricks unifies data, AI, and governance across financial crime, banking, and wealth-management workflows, and connect with leaders at Sibos 2026 in Miami.
A practical approach to end-to-end Solvency II reporting in Databricks
Databricks now supports Solvency II end to end, connecting data ingestion, reserving, capital calculation, governance, and disclosure into a single workflow instead of fragmented systems and teams. A demo shows how this unified approach delivers one control view, governed automation, AI-assisted review, and faster scenario analysis for reporting readiness.
Beyond embedding: How to secure AI/BI Dashboards for every viewer
A single AI/BI dashboard can now serve every viewer securely: one entitlements table plus the signed embed token's __aibi_external_value control row-level access per viewer, so teams avoid duplicating dashboards or repeating filters across queries. Access is driven by identity-provider group membership rather than manually maintained user lists, and the pattern layers default-deny protections—masked columns, refused tokens for unentitled viewers, and Unity Catalog row filters—for defense in depth.
Introducing Consort: Test-driven development on a branching database
Consort brings test-driven development to database branching, giving each branch its own isolated copy of data so tests run against realistic state without touching production. The post walks through what changes when your database can branch like code, demonstrates an agentic development workflow built on it, and shows how to try Consort yourself.