MLflow
Recent items mentioning MLflow across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
What is MLflow?
MLflow is an open source platform for managing the lifecycle of machine learning models and, increasingly, LLM applications and agents. It tracks experiments (parameters, metrics, artifacts, code versions), packages models and agents in a standardized format, and registers them for versioned deployment. Databricks offers a fully managed and hosted version that builds on the open source experience, with the Model Registry integrated into Unity Catalog.
The problem it solves is losing track of what produced what. Without lifecycle tracking you can't reproduce a model, compare candidate versions, or explain an agent's behavior in production. MLflow Tracing records the inputs, outputs, and metadata of each intermediate step an application takes, which turns debugging and auditing from guesswork into reading a trace.
MLflow 3 moved the project's center of gravity toward generative AI: observability, evaluation with built-in LLM judges, and prompt management for agents and LLM apps, plus new concepts like Logged Models and Deployment Jobs. On Databricks, the managed version adds enterprise features such as encryption with customer-managed keys.
Is MLflow free?
The open source project is free to self-host. On Databricks you get a fully managed and hosted version that builds on the open source experience and adds enterprise security, disaster recovery, encryption with customer-managed keys, and tight Unity Catalog integration.
What's new in MLflow 3 compared to MLflow 2?
MLflow 3 centers on GenAI: end-to-end tracing for agent-based systems, evaluation with built-in LLM judges for safety, relevance, and correctness, and centralized prompt management with versioning and Unity Catalog integration. It also introduces the LoggedModel entity for organizing and comparing model variants, plus Deployment Jobs, with automatic tracing integrations for many popular GenAI libraries and frameworks.
Is MLflow only for classical machine learning?
No. MLflow Models is a standardized format for packaging both machine learning models and AI agents, and MLflow Tracing records the inputs, outputs, and metadata of each step a GenAI application takes. Databricks positions MLflow evaluation and monitoring for agents and ML models alike.
How does MLflow relate to Unity Catalog?
On Databricks, the MLflow Model Registry is integrated with Unity Catalog, so models are versioned and governed alongside your other data and AI assets. That gives you centralized governance and model lineage tracking, plus the ability to access models across workspaces and discover them for reuse.
Sources: MLflow on Databricks (Databricks docs) · MLflow 3 for GenAI (Databricks docs) · MLflow 3 release notes (mlflow.org)
MLflow 3.18.0 introduced automatic issue detection, a redesigned evaluation and trace comparison UI, an experimental Skill Registry, and online code-based evaluation 6. Concurrently, Anthropic's Claude Platform rolled out native OpenTelemetry trace exports directly into Databricks MLflow and Unity Catalog 7, alongside new MLflow AI Gateway capabilities for enforcing guardrails and budget policies across third-party coding agents 8.
Generated daily from the 10 most recent items mentioning MLflow. Click any [N] to jump to the source.
Tabular foundation models on Databricks AI Runtime
Tabular foundation models accelerate predictive ML workflows by generating predictions on new datasets without requiring task-specific training. Databricks AI Runtime enables teams to benchmark and fine-tune these models against existing baselines using managed GPUs, MLflow tracking, and Unity Catalog integration.
NewsDistill a 120B LLM into a 0.8B Model on Databricks
LLM distillation teaches a smaller student model on answers from a larger teacher model using a narrow task at high volume, saving cost and improving speed while maintaining accuracy. This Databricks tutorial uses Foundation Model APIs, Serverless GPUs, MLflow evaluate, and Model Serving to build and deploy the smaller model end to end.
Deploy models pattern via MLflow deployment jobs + multiple workspaces
Hi, I'm quite new to MLOps and need some hints for setting up what Databricks calls "Deploy model" pattern ( https://docs.databricks.com/aws/en/machine-learning/mlops/deployment-patterns ). Our Data Science team currently trains ML models in dev environment, in our dev Databricks account/workspace. They have direct access to production data, and the trainingㅤ is quite expensive. We'd like to train the model once and then promote it to production, which lives in a separate Databricks account/workspace. My idea to handle the deployment was to use MLflow deployment jobs ( https://docs.databricks.com/aws/en/mlflow/deployment-job ) and just allow the Data Science team to register the model in the production Unity Catalog. After the registration, the deployment job would kick in, do whatever else is needed, and then stop at an approval step. However, it turned out they can register the model version in the Unity Catalog from the dev workspace, but the deployment job doesn't run automatically. That makes the whole process suboptimal, as they now have to ping me to run the deployment job manually each time they register a new model version. I did some testing and the sole reason of that behavior seems to be the fact that the model comes from a different workspace, which doesn't "see" the attached deployment job that lives in the production workspace. Am I missing something here, or is it designed to work like that? What are the alternatives? submitted by /u/AvaPL [link] [comments]
Databricks Asset Bundles now deploy registered model aliases to Unity Catalog and resolve bugs that repeatedly reapplied permissions and grants to recreated resources. The CLI also restores Databricks-specific pip magic execution in Jupyter notebooks over SSH and removes the CRITICAL compute priority class from AI Runtime.
How I check every prompt before the LLM on Databricks: GovTech's LionGuard on a small CPU endpoint
I wanted one shared safety check that every agent, job or SQL query on Databricks calls before the LLM. How it works: - LionGuard 2 Lite from GovTech Singapore: a small classifier on Google's EmbeddingGemma. It scores insults, hate, violence, self-harm and misconduct, and it handles English, Singlish, Chinese, Malay and Tamil. - Packaged as one MLflow model, registered in Unity Catalog, served on a small CPU endpoint. - From SQL, ai_query sends each question to the guard. Only questions with an unsafe score under 0.5 go to the LLM. Results on 7 synthetic questions: a Singlish insult, a Malay threat, a harmful Chinese message and an English "how do I hack the website" question were blocked. Safe questions in English, Singlish and Tamil were answered. Lessons: - The full set-up took 25 minutes. 19 of them were the endpoint start. - Do not use scale to zero for a production guard (cold start). - Turn on inference tables from day one, so you can show what was blocked. - It does not detect prompt injection. That needs a separate check. How do you check prompts for your agents today? Built-in AI Gateway guardrails, a custom endpoint, or both? submitted by /u/sundarleads [link] [comments]
MLflow 3.18.0
MLflow 3.18.0 adds automatic issue detection, a redesigned evaluation and trace comparison UI, an experimental Skill Registry, and experimental online code-based evaluation that requires a self-hosted server with job containers. Breaking changes include a new search_logged_models sort order, no registered_model_meta in models:/ downloads, and case-sensitive MCP version identity on MySQL and SQL Server, so back up the database and run mlflow db upgrade before upgrading.
Claude Platform now supports native OpenTelemetry observability in Databricks MLflow
Claude Platform's native OpenTelemetry export is now in Public Beta, sending traces from Anthropic SDKs, the Messages API, and Claude Managed Agents into Databricks MLflow, where they're stored in Unity Catalog. Use trace analysis and MLflow Issues to investigate bad answers and spot repeated tool calls, then have domain experts review answers and rerun evaluation on the same test cases to check whether changes improve quality and cut unnecessary work.
Governing Coding-Agent Sprawl with MLflow AI Gateway
The MLflow AI Gateway puts one governed endpoint in front of coding agents like Cursor, Claude Code, Codex CLI, Gemini CLI, and Pi. It routes each request by endpoint name, applies PII and safety guardrails and a budget policy, and traces every call with token counts and agent, provider, and endpoint tags. A runnable demo on MLflow 3.17.1 shows per-provider attribution and blocked bad requests, along with the latency cost of LLM-judge guardrails.
ACTUALLY USEFUL Big Data Project Ideas
Heyy everyone! We have a Big Data class project proposal coming up, and I want to make an actually useful project (something challenging, or something that could sell, or something that solves an actual problem people have). Not yet another fancy showpiece class project. Requirements: The dataset must be at least 10GB, be genuinely large-scale and require distributed processing The project must include some or all (or even more): data warehouse (snowflake/redshift/any other) airflow elt/dbt spark, sparkml mlflow kafka/spark structured streaming/flink nosql, vectordb llmops Practitioners, help me with some ideas? (ChatGPT and Claude didn't help much). submitted by /u/mulberry-cream [link] [comments]
Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)
Log custom PyFunc models with MLflow using python_model
I work at Abilytics (data and AI consulting), so I'm not a neutral party here. If standard MLflow flavors do not fit your custom inference pipeline, subclass mlflow.pyfunc.PythonModel and override the load_context and predict methods. This ensures any custom deserialization logic or specialized third-party library dependencies execute safely within the worker environment during batch scoring or serving. When logging the model via mlflow.pyfunc.log_model, always supply the artifacts dictionary parameter to bundle necessary configuration files or tokenizer states directly into the artifact store. This prevents missing file errors on the scoring cluster because MLflow packs those dependencies into the model directory structure automatically. Call the model back downstream using mlflow.pyfunc.load_model with the runs:/ URI scheme to execute distributed inference via Spark UDFs without rewriting code for production clusters. submitted by /u/AbilyticsEng [link] [comments]
From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry
Improve ai_extract accuracy by matching the fix to the problem: schema descriptions for field-level rules, prompts for document-level logic, precision mode for structural complexity, and citations to diagnose errors. Version prompts and schemas in MLflow Prompt Registry, deploy or roll back via aliases, and compare candidates against human-reviewed ground truth before promotion, with Genie Code automating much of this eval-and-fix loop.
How to teach with Databricks Free Edition
Databricks Free Edition can anchor a full data and AI curriculum, from intro data science and statistics through data engineering, BI, machine learning, AI, and capstone projects. Students work with the same tools used in industry, including Apache Spark, Delta Lake, MLflow, Lakebase, Genie, and Databricks Apps, while instructors can hand out assignments through an existing LMS or GitHub and share datasets via cloud storage, Open Sharing, or Marketplace.
NewsTrain and fine-tune models faster with AI Runtime
Databricks AI Runtime is a managed GPU experience for distributed training and fine-tuning of AI models at scale. The video demonstrates using AI Runtime to train and fine-tune a model and then analyze the run to improve model quality and efficiency.
MLflow 3.17.2
MLflow 3.17.2 records dataset, scorer, and agent lineage on runs created by mlflow.genai.evaluate. It also fixes MlflowSparkStudy so executors reseed the sampler and forward batch settings, and stops MlflowStorage from marking Optuna trials finished before their values are saved.
TutorialsBuilding Production-Ready AI Agents on Databricks | Agent Bricks Demo
Building production ready AI agents on Databricks involves a six step workflow spanning vector search indexing, agent as code, MLflow tracing, Unity Catalog registration, evaluation, and deployment with review apps. MLflow tracing and evaluation metrics help monitor agent execution steps, relevance, safety, and topic adherence in real time.
Zepto Scales Customer Support on Databricks and MLflow
5 Simple Tips to Supercharge Your Claude SDK Experience with MLflow
Claude SDKs now emit OpenTelemetry traces natively, allowing you to seamlessly monitor and evaluate your agent runs with MLflow. Learn five practical tips to capture unified end-to-end traces, track token usage, score executions, and compare performance across different prompts.
How our AI Research team uses AI Runtime
Databricks AI Research uses AI Runtime to train and fine-tune models across hundreds of GPUs, supporting an 8x surge in training jobs fueled by agent-driven research. Built-in GPU reliability, fast job launches, and native integrations with MLflow and Unity Catalog accelerate these automated experimentation workflows at scale.
How to trace Claude agents via Native OpenTelemetry Export
Claude now natively emits OpenTelemetry traces across Claude Managed Agents, the Messages API, the Claude SDKs, and Claude Code. Learn how to configure the export for each product using open-source MLflow as your trace backend.
NewsBuilding on Lakebase | Episode 2: Anatomy of an AI app: build the operational + agentic app
This video demonstrates the architecture and deployment of a customer support AI co-pilot built as a Databrics app backed by Lakebase. It walks through configuring a transactional operational store, hybrid search, managed agent memory, and MLflow tracing to power an interactive agentic application.
MLflow 3.17.1
MLflow 3.17.1 introduces the mlflow.genai.databricks.review_queues module for managing review queues on Databricks and adds Decision API support to make_judge(). It also fixes Claude Code OpenTelemetry trace rendering in the UI and ensures Databricks evaluation feedback correctly records the latest scorer version.
NewsAgent Bricks: The Agent Developer Platform | Overview
Databricks introduces Agent Bricks, a developer platform that provides infrastructure components for building, deploying, and scaling AI agents. The platform features flexible hosting, a unified API for model routing, secure sandboxes, managed memory, and tools for real-time tracing and evaluation.
From post-training to production: How TinyFish runs its web-native model on Databricks AI Runtime
TinyFish established a continuous path from post-training to deployment by using Databricks AI Runtime on managed GPUs, MLflow for evaluation, and Databricks Model Serving for production inference. Tested across 2,178 web-agent tasks, the resulting web-native model achieved a 77.6% task-success rate in internal go/no-go benchmarking.
MLflow 3.17.0
MLflow 3.17.0 introduces fine-grained access control for runs and traces, links evaluation metrics directly to managed datasets, and adds support for Databricks Unity Catalog model service resources. Self-hosted SQL tracking servers require a coordinated schema migration via mlflow db upgrade with all writers stopped, as rolling deployments are not supported.
How to fit ~1GB+ embedding model into a 2Gi Kubernetes pod? Getting OOMKilled
Hi, Deploying a FastAPI to Kubernetes that uses a multilingual sentence-transformers embedding model (ONNX backend, CPU only). My source data lives in a Delta table in Databricks, and the app reads from it and generates embeddings. The app takes user input (text) at embeds it, and compares it against stored embeddings for semantic similarity. So at least the query embedding has to happen live. Pod resources: - CPU: 1 request / 2 limit - Memory: 2Gi request = 2Gi limit (platform policy requires memory request and limit to be 1:1) Current docker setup: - Multi-stage Docker build (python:3.11-slim) - CPU-only PyTorch - The model is downloaded at build time, so it's baked into the image Image breakdown: - HF model cache: ~1.1 GB - torch: ~650 MB - pyarrow, scipy, transformers, pandas: ~100–150 MB each - Plus sklearn, onnxruntime, mlflow, and others The pod gets OOMKilled at startup or shortly after. With a ~1GB model, torch, pandas/pyarrow, and the ONNX runtime session all in one process, I think I'm just over 2Gi. What I'm trying to figure out on where do you store large models in production? Would love to hear what setups have worked for you. Thanks! submitted by /u/runningnozone [link] [comments]
EventsDemo: Building a Governed AI Agent with Unity AI Gateway
This video demonstrates how to build, update, and govern a store operations AI agent using Databricks Agent Bricks and the Unity AI Gateway. The tutorial highlights integrating custom Model Context Protocol servers, recording execution traces with MLflow, and enforcing security policies and budget controls.
MLflow traces accepted by StartTraceV3 (200 OK) but never stored: GetTrace returns NOT_FOUND
The CLI introduces a preview image push command for the Databricks Artifact Registry, preconfigures coding assistants in serverless SSH sessions, and adds expiration flags for Postgres branch creation. In Asset Bundles, clusters now support library configurations, Terraform deployments migrate to the direct engine prior to execution, and Postgres snapshot schedules are restricted to YAML.
Workshop on bringing systematic evaluation and MLflow tracking to LLM apps, Oct 3
If you're already using MLflow for experiment tracking on the ML side, there's a decent chance your LLM work hasn't caught up to the same standard yet. Most teams are still hand-tuning prompts and eyeballing outputs while everything else in the stack gets versioned and tracked properly. Serj Smorodinsky and Brett Kennedy, co-authors of a book on LLM applications, are running a live 3-hour workshop on Oct 3 that applies that same rigor to LLM development. What's covered: Programming LLM behavior with DSPy signatures and modules instead of hand-written prompt strings Building a baseline classifier live, from a real task Constructing an evaluation dataset with task-specific metrics so "better" is measurable, not a feeling Reading failure patterns directly out of the eval results Few-shot and instruction-level optimization applied systematically on top of the baseline Experiment tracking and LLM trace management through MLflow, so every run is reproducible and comparable Saving and reusing optimized DSPy programs across projects Communicating LLM reliability to stakeholders, which usually gets skipped entirely It's basically the MLflow discipline this community already applies to models, extended to prompts and LLM pipelines. Aimed at people already past "which model should I use" and into "how do I make this reliable." Full details here. submitted by /u/camerongreen95 [link] [comments]
Can Jev replace your LLM judge? Part 2: Testing harder answers
TypeSafe's Jev model evaluates MLflow QA answers roughly seven times faster and significantly cheaper than larger LLMs, but it misses subtly incorrect technical details such as reversed roles and omitted permissions. Ahead of Jev integration arriving in MLflow 3.17, practitioners can use its confidence probabilities to fast-track clear verdicts while escalating borderline cases to a larger model for review.
Laya off the benchmark: can a zero-shot decision model route real SQL traffic?
Weekend-ish experiment on Databricks. One table, TPC-H orders, 15M rows, living in two places at once: Lakebase (Databricks' managed Postgres), a continuously synced copy with a btree index on the key. Sub-100ms point lookups, useless for a GROUP BY over 15M rows. Delta behind a serverless SQL Warehouse. Great at scans and aggregations, slow at fetching one row. The synced table is the nice part: native continuous Delta to Lakebase Postgres (needs a PK + CDF), so it's one dataset under one Unity Catalog, replication handled by the platform instead of a homemade pipeline. Then I put laya (convaiinnovations/laya, a non-autoregressive zero-shot decision model, vanilla, no fine-tuning) behind a FastAPI Databricks App. It reads each query and picks the engine. MLflow traces input to decision to execution. I submitted by /u/Limp-Park7849 [link] [comments]
AI Runtime commands have moved from experimental to databricks air, and SSH commands now support keeping detached background processes running after tunnel disconnect. Databricks Asset Bundles direct engine resolved multiple issues around unnecessary resource recreations, Unity Catalog grant convergence, and Git-sourced Python tasks.
MLflow 2.11.5
Users can now opt in to route Unity Catalog model registry artifact uploads and downloads through the Databricks SDK Files API. This update provides an alternative transfer mechanism for managing model artifacts within Databricks environments.
Can Jev replace your LLM judge? Evaluating quality, cost, and latency
Benchmarking Jev against GPT, Claude, and DeepSeek using MLflow reveals how it measures up as an alternative LLM judge. Review side-by-side performance across quality, cost, and latency to see if Jev can replace your current evaluation models.
NewsHow adidas Uses Databricks to Build Better Products
Adidas uses Databricks' lakehouse platform to centralize all its data—from product to football-related insights—enabling faster analytics across the organization. The company's Genie analytics tool helps analysts spend less time processing data and more time on strategic questions, ultimately supporting better product development.
Building Custom Agents on Databricks:LangGraph, Atlan-Grounded Routing, and Per-Run Cost with MLflow
MLflow 3.16.1
MLflow 3.16.1 removes the default basic-auth admin password from basic_auth.ini as a security fix (GHSA-gq3w-7jj3-x7gr) and adds support for span links in Unity Catalog traces. The release also fixes trace location resolution in Databricks Model Serving, includes a timeout option for the @scorer decorator, and improves web UI access control for non-admin users.
Could a “Data → Agent” composer be useful for Databricks?
I've been thinking about a gap between Databricks data and agent frameworks. Databricks already has a lot of the building blocks: - Unity Catalog - Genie / Genie Agents - MCP - Vector Search - AI Gateway - Agent skills/tools - MLflow - Omnigent / Kasal And tools like Omnigent and Kasal already solve a lot of the agent orchestration/execution side. But I'm wondering about the step before that: What if a customer already has a large, curated and governed Databricks data estate — how do we turn that data estate into an agent-ready configuration without manually wiring everything together? Something like: Existing Databricks Data Estate ↓ Data-to-Agent Composer ↓ ┌────────┼─────────┐ ↓ ↓ ↓ Domains Semantics Metrics ↓ ↓ ↓ Genie MCP Skills └────────┼─────────┘ ↓ Agent Configuration ↓ Omnigent / Kasal ↓ Agent The idea wouldn't be to build another chatbot or another agent framework. It would be a Databricks-native composition/bootstrapping layer that understands an existing Unity Catalog/data estate and generates the pieces needed for agents to work with that data — domain boundaries, semantic context, approved tools, Genie configuration, MCP exposure, skills, policies, evaluation setup, etc. In other words: Kasal/Omnigent: Agent → Tools/Data Proposed layer: Data Estate → Agent I'm curious if this is already solved somewhere in the Databricks ecosystem, or if people are currently doing this manually when building enterprise data agents. Would love to hear how others are approaching the “existing data estate → production-ready data agent” problem. submitted by /u/imsuryya [link] [comments]
Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow
Zepto built its customer support agents on Databricks and MLflow using an evaluation-first approach, treating traces, golden datasets, and LLM-as-judge checks as core infrastructure and connecting development and production through a dual-loop architecture with a strict quality gate. The result: 65% lower support costs and payback in under a month, giving other AI builders a reusable blueprint for engineering reliability, cost, and risk into production-grade agents.
TutorialsHow to Build and Serve Production ML Features | Databricks Feature Store Demo
Databricks Feature Store provides a unified feature views abstraction for batch and streaming data with automatic point-in-time joins for training and built-in MLflow experiment tracking. The same feature definitions deploy directly to production with online serving on Lake Base, eliminating the need to rewrite features while enabling model and feature serving to scale together.
Building Custom Agents on Databricks:LangGraph, Atlan-Grounded Routing, and Per-Run Cost with MLflow
MLflow 3.16.0
MLflow 3.16.0 makes the redesigned trace explorer the default interface, introducing natural-language custom trace views via the MLflow Assistant, session grouping for multi-turn conversations, and span links. The update also adds Unity Catalog model service support for built-in evaluation judges, per-user AI Gateway budget policies, and fail-closed authorization by default.
Difference between Workspace and Unity Catalog experiments when using MLflow autologging?
How are you actually setting up AI/LLM evals in Databricks end-to-end? Looking for a step-by-step production workflow
We have a product where Databricks is our backend , and we’re now trying to properly evaluate and improve the quality of the AI-generated answers in our application. I’m looking for advice from people who have actually implemented LLM/GenAI evaluations in Databricks in production . I’d really appreciate an end-to-end, step-by-step explanation of how you would set this up from scratch. Specifically, how would you approach: Define what a “good answer” means Accuracy / correctness Relevance Completeness Groundedness / faithfulness to our data Hallucination rate Citation/source correctness Following user instructions Response consistency Latency and cost Create a benchmark / golden dataset Should we manually create a set of representative user questions? How many questions are enough to start? Should each question have an expected answer? Should we store expected SQL/results, expected sources, or just an expected natural-language response? Where should this benchmark dataset live in Databricks? How do you keep it updated as the product evolves? Set up automated evaluations What Databricks/MLflow tools should we be using today? MLflow evaluation? LLM-as-a-judge? Custom scorers? Human evaluation? How do you combine these rather than relying on one score? Evaluate RAG / data-grounded answers Our AI answers questions based on enterprise data in Databricks. How do you separately evaluate: Retrieval quality Whether the correct tables/documents were selected Context relevance Groundedness Final answer correctness Whether the model invented something that wasn’t in the retrieved data Evaluate text-to-SQL / analytics questions If a user asks something like: “What was average building occupancy last month?” should we evaluate: Generated SQL Tables selected SQL execution result Final natural-language answer separately? What is the recommended architecture for this? Guardrails Where should guardrails sit in the architecture? For example: Prevent hallucinated numbers Prevent querying unauthorized tables Detect PII Prevent prompt injection Enforce tenant/user permissions Block unsupported questions Force answers to cite their source Return “I don’t know” when confidence is too low Should guardrails be part of evaluation, inference, or both? Production monitoring Once this is live, what should we log for every AI request? For example: User question → retrieved context → generated SQL/tool calls → query result → final response → model → prompt version → latency → tokens → cost → evaluation scores → user feedback Is that roughly the right model? Regression testing When we change: System prompt Model Retrieval strategy SQL generation logic Tools Temperature Data sources how do you automatically run the benchmark again and determine whether the new version is actually better? Do you set minimum score thresholds before allowing something to deploy? Human feedback How are people incorporating thumbs-up/down or analyst review into their evaluation datasets? Do production failures automatically become new benchmark cases? Making answers more precise This is ultimately my main goal. If our AI currently gives an answer that is “mostly correct,” what is the systematic process for figuring out why it isn’t fully correct? Is the best workflow something like: Production traces → identify failure → categorize failure → add to benchmark → improve retrieval/prompt/tool → run eval → compare against baseline → deploy → monitor Or is there a better approach? I’m especially interested in what the ideal Databricks-native architecture looks like: User Question → Agent / LLM → Retrieval / SQL / Tools → Databricks data → Response → MLflow tracing → Automated evaluators → Benchmark dataset → Regression testing → Production monitoring If you’ve implemented something like this, I’d love to know what you would build first, second, third, etc. Even a practical example like: Week 1: create benchmark Week 2: add tracing + scorers Week 3: add regression tests Week 4: add guardr […truncated]
MLflow 3.15.2 introduces immutable evaluation dataset versions and a scorer_ensemble primitive for combining scorer results. The patch also fixes a Databricks telemetry deadlock and addresses issues in MemAlign aligned judges and runs.status constraint handling.
EventsBuilding Governed Agents with Databricks
Databricks extends its Unity Catalog governance layer to AI agents, MCP servers, and tools to solve "agent sprawl" by providing centralized discovery, access controls, and audit logging. The system allows developers to quickly build agents while IT teams enforce fine-grained policies, manage credentials, and track end-to-end request and response lineage.
NewsAI Runtime CLI | Serverless GPU LLM Training
Databricks AI Runtime is a CLI tool that enables distributed LLM training on serverless GPUs using YAML configuration files, supporting deployments up to 256 H100 GPUs with built-in MLflow experiment tracking and hardware metrics. Users develop training code locally in their preferred environment and submit jobs via CLI commands that automatically handle infrastructure provisioning, log streaming, and performance monitoring.
NewsHow AI and Data Keep 2.3 Million Lawns Healthy | TruGreen & Databricks
TruGreen uses Databricks Genie and Lakehouse to manage 2.3 million lawns with AI that optimizes service timing and predicts customer churn using weather, soil, and service data. The system enables non-technical branch managers to take daily actions through customized reports without requiring data expertise.
NewsDatabricks News: ZeroOps, DABs, Indexes, Genie, sandboxes, migration from PowerBI, secrets
Zero Ops automatically detects errors in jobs and data quality with lineage analysis and proposes code fixes, while DABs now default to direct mode instead of Terraform with automatic state migration. Full-text search indexes deliver 400x faster queries on billion-row tables, Genie automatically converts PowerBI dashboards to Databricks metric views, and Unity Catalog secrets support granular read and reference-only permissions.
Job definitions now support trigger configuration and MLflow artifact location for AI runtime tasks. Pipeline connector options and GCP endpoint settings have been expanded with new configuration fields.
The SDK now supports job triggers and MLflow artifact locations in the Jobs API, enabling more flexible job scheduling and artifact management. New connector configuration options are available for Pipelines, and GCP endpoint settings have been expanded.
NewsBuilding Agents on Databricks with Custom Apps and Omnigent
This video demonstrates how to build, update, and govern custom AI agents on Databricks using Agent Bricks, Databricks Apps, and Omnigent. The tutorial shows how to integrate Model Context Protocol servers, track execution with MLflow traces, schedule automated agent tasks, and manage security policies through Unity AI Gateway.
TutorialsBuilding Agents on Databricks with Custom Apps and Omnigent
The video demonstrates how to build, update, and govern a store operations AI agent on Databricks using Model Context Protocol servers and custom apps. It shows how to use Omnigent and CodeX to add new context and tools, redeploy the application, and manage governance and traces through the Unity AI gateway.
NewsAI for Mental Health: Crisis Text Line and Databricks
Crisis Text Line provides 24/7 text-based mental health support to millions of individuals facing crises like anxiety, bullying, and financial hardship. The organization uses the Databricks platform to safely analyze anonymized conversation data and improve how researchers understand and serve people in need.
MLflow 3.15.1
MLflow 3.15.1 fixes version parsing issues on Databricks Serverless and corrects env_pack behavior on ARM client images. Scorer versioning documentation has been clarified.
Evaluating and Improving Agent Skills with MLflow
MLflow enables evaluation-driven development for AI agent skills by combining traces, evaluation datasets, custom evaluators, and experiment tracking to treat agent capabilities like testable software components. Instead of grading only final outputs, teams can use execution traces to systematically measure critical behaviors such as tool sequencing, policy compliance, and execution efficiency across skill iterations.
MLflow 3.15.0 introduces an MCP Registry for registering and sharing Model Context Protocol servers, enhances the Assistant with multi-provider LLM support and per-session token usage tracking, and enables proxy-less artifact transfers via presigned URLs to reduce server load and timeouts on large files. Additional improvements include sharable Runs table views, multi-modal image attachments for LLM judges to evaluate vision tasks, and numerous bug fixes across tracing, evaluation, gateway, and UI components.
Workspace ID validation for unified-provider resources shifted from plan time to apply time, reducing unnecessary API calls and eliminating false positives for restricted credentials and dynamic workspace IDs. The release adds a databricks_recipients data source for Delta Sharing, trace_location support for MLflow experiment traces in Unity Catalog, and fixes for VIEW column comment updates and access control rule set drift detection.
Get Tuesday's version of this
Tracking MLflow? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.
