Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Summary
Databricks now supports prompt caching for open-source models across all workloads, automatically accelerating LLM inference by reusing repeated prompt prefixes. This feature boosts throughput by 2.5x and reduces P50 latency by 3x for models like GPT-OSS, with no setup required.
Summary generated by brickster.ai. For the full article, follow the source link above.
More from Databricks Blog
Pharma launch analytics: How to compress the first 90 days and win the three years that follow
Databricks Genie for Commercial Launch Intelligence helps pharma companies compress 90 days of launch analytics into immediate insights. This enables commercial leaders to quickly interrogate launch data, make weekly decisions, and drive long-term growth.
Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering
Octopus Energy achieved a 50x cost reduction in their margin data engineering pipelines by re-architecting on Databricks for UK MHHS regulation. They leveraged Delta Lake Change Data Feed and Databricks Serverless to process 48x more data at a fraction of the original cost, improving freshness from weekly to daily.
Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks
Databricks now supports writing OpenTelemetry traces directly to Unity Catalog tables via a fully managed, serverless ingestion path. This enables governed, analytics-ready observability data with long-term retention and unified evaluation workflows, without operating OTel infrastructure.
How World Bank Group uses databricks to eradicate poverty through shared knowledge
The World Bank Group built a unified data and AI platform on Databricks, leveraging Unity Catalog, Volumes, Genie, and AI Gateway to connect structured operational data with unstructured documents. This eliminated manual research bottlenecks, enabled natural language access to trusted insights, and now supports millions of document downloads monthly to accelerate global knowledge sharing for poverty reduction.
How Databricks Genie democratizes data access in financial services
Databricks Genie now democratizes data access for financial services business leaders by enabling natural language querying of governed data. This eliminates the "Last Mile of Data Democratization" by removing the need for SQL skills or BI tool training.
Using observability data to prevent incidents
Databricks Genie enables natural language querying of telemetry data, allowing leaders to proactively identify risks and shift from reactive firefighting to reliability intelligence. This helps engineering teams move beyond optimizing for response time (MTTR) to prevent incidents by surfacing upstream reliability risks that impact revenue, roadmap velocity, and customer trust.