Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Summary
Databricks now supports prompt caching for open-source models across all workloads, automatically accelerating LLM inference by reusing repeated prompt prefixes. This feature boosts throughput by 2.5x and reduces P50 latency by 3x for models like GPT-OSS, with no setup required.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Inbound Private Link now supports account-level Genie One, the account console, and custom URLs
Enterprises adopting Databricks for their most sensitive data commonly rely on Inbound...
Busting SQL Migration Myths: How New SQL Features Make Lift-and-Shift to Lakehouse Easier
New Databricks SQL Scripting features enable direct lift-and-shift migrations of legacy PL/SQL, allowing teams to translate procedures, cursors, temp tables, and transactions line for line without rewriting business logic. Once migrated, these procedures retain their original control flow while gaining built-in Unity Catalog lineage and access control.
Designing effective Genie Agents from a single prompt
Genie Agents let teams turn trusted business context into domain-specific AI agents from a single prompt, reasoning across structured data, documents, and files governed by Unity Catalog without constant prompt tweaking. The post argues that weak agent answers usually stem from missing context rather than bad prompts, and recommends starting with one focused, benchmarked use case as the foundation for expanding with more knowledge, tools, and workflows.
Databricks Document Intelligence: pushing the frontier for complex document extraction
Databricks Document Intelligence's new Precision Mode targets the failure points of complex document extraction—cross-page reconciliation, long outputs like thousands of invoice line items, and schemas requiring reasoning and computation—by pairing custom finetuned extraction models with an agentic harness that reasons in stages and merges parallel subagent outputs. Across six complex document benchmarks, it outperforms the next best frontier model by seven points on accuracy.
When it comes to Governance, Retailers need a control plane for context
Databricks now gives retailers a single governed layer for AI, so every model, agent, and application draws on the same trusted data and cost controls instead of each team standing up its own. As retail AI extends into stores, merchandising, and daily decisions, this control plane lets retailers put sensitive data like purchase history and loyalty behavior to work for personalization while keeping tight control over who can access it and how.
Evaluating AI Agents Live at the Grounded Reasoning Cup
Stanford's team won the Grounded Reasoning Cup with 63.3% accuracy on OfficeQA Pro V2, a new 120,000-page U.S. Treasury document benchmark, using an end-to-end agent optimization approach combining reusable skills, document-representation fallbacks, and adaptive verification. Across all 11 academic teams, out-of-the-box frontier agents averaged under 30% accuracy, showing that agent performance tuned on one benchmark doesn't reliably generalize to a new corpus.