Reducing Qwen training time by 6x on AI Runtime
Summary
Qwen fine-tuning can be accelerated by 6x on AI Runtime by installing the correct kernels and improving the data pipeline, even when baseline GPU utilization already reaches 98%. AI Runtime enables you to diagnose these performance issues using a standard torch trace of an LLM.
Summary generated by brickster.ai. For the full article, follow the source link above.
Topics
More from Databricks Blog
Sizing Lakebase Search: lessons from a real app and 4 million passages
Sizing the Local File Cache to match the vector index keeps up to 98% of pages in memory and drops warm vector latency to single-digit milliseconds directly inside Lakebase Postgres. Reducing the search candidate pool to 200 cuts page reads by two-thirds while preserving 90.5% of top-10 results, with an always-on 0.5 CU floor providing an optimal production baseline for just $31 to $40 per month.
Unlock powerful insights and forecasts with Predictive AI Functions
Databricks has introduced five Predictive AI Functions, featuring an upgraded ai_forecast alongside the new ai_top_drivers, ai_detect_anomalies, ai_predict_value, and ai_predict_class. These tools allow SQL analysts and business users to generate predictions, detect anomalies, and uncover metric drivers directly within Genie and existing workflows without building or maintaining models.
Power real-time applications and agents with Document Intelligence
Document Intelligence real-time REST APIs enable applications to parse, classify, and extract information from documents as they arrive rather than relying on delayed batch pipelines. You can embed these APIs directly into Databricks Apps or custom agents to power interactive workflows and drive real-time business decisions.
Introducing Private Network Gateway
Databricks has introduced Private Network Gateway (PNG), a managed service that connects serverless workloads to your private networks and resources through a single network-level configuration. Utilizing your existing routes, DNS, and firewall rules, PNG is now available in Public Preview on Azure and Private Preview on AWS.
A practical guide to real-time analytics in 2026
Rather than relying on bolted-on serving layers that duplicate data and fragment governance, a lakehouse delivers millisecond query performance and high concurrency directly on governed data. Learn how to evaluate workloads across freshness, latency, and concurrency, alongside best practices and industry use cases for scaling real-time analytics.
What's New in Lakeflow Designer - October 2026 Roundup
Lakeflow Designer now includes new operators for data cleaning, deduplication, quality checks, and flexible joins, along with canvas support for materialized views, table merges, and live cursor presence. The update also previews upcoming additions, such as Databricks Apps integration, full-output data profiling, and native migration tooling with Lakebridge.