MLflow

Multi-Harness AI Agents Need Multi-Layer Observability: Omnigent in MLflow

Omnigent unifies multi-harness agent orchestration and now delivers automatic observability across every agent with MLflow Tracing, requiring no code changes. This post details how Omnigent in MLflow provides multi-layer observability for multi-harness AI agents.

MLflow 3.14.0 introduces new GenAI features including agent onboarding, durable tracing for Claude Code, review queues for traces, and a revamped evaluation dataset UI. It also includes a new LLM Playground for prompt iteration and changes the default serialization format for several MLflow model flavors to `skops` or `pt2`.

How to Manage your LLM Teams using MLflow's Role-Based Access Control

MLflow's new Role-Based Access Control (RBAC) helps LLM teams define reusable roles, isolate workspaces, and enforce fine-grained permissions across prompts, experiments, and AI Gateway resources. Learn how to manage your LLM teams using these new MLflow RBAC capabilities.

How Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude

Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude, converting 700-page FDA manuals into real-time answers for frontline staff using Foundation Model APIs and cutting compliance report compilation from two weeks to under two minutes. The solution, a native Databricks App with Lakebase Postgres and Unity Catalog, unifies nine siloed data sources and employs a multi-agent orchestration framework with Judge LLMs and MLflow tracing for personalized, continuously refined intelligence.

MLflow 3.13.0 introduces a new Role-Based Access Control system with an Admin UI for managing users and permissions, alongside trace retention and auto-archival to object storage. This release also includes one-click observability for coding agents, new engines for MLflow Assistant, and an official Helm chart for Kubernetes deployments.

Route Claude Code Through MLflow AI Gateway

MLflow AI Gateway now supports routing Claude Code, providing full observability, budget controls, and guardrails for all your coding agent sessions. This integration requires no changes to your existing Claude Code usage.

MLflow now features a major overhaul of Role-Based Access Control with a new Admin UI and unified permission APIs. It also introduces end-to-end trace archival, Helm charts for Kubernetes deployment, and a new API for stress-testing GenAI agents.

From "What Happened?" to "What Will Happen?"

Conversational BI now delivers predictive answers in seconds, not days, by fusing Genie for dynamic feature engineering with TabPFN for zero-training prediction, orchestrated by Agent Bricks. This self-assembling pipeline eliminates data science bottlenecks for business users, providing a governed experience backed by Unity Catalog and MLflow.

Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code

MemAlign, an open-source MLflow framework, significantly improved the evaluation of traditional machine learning in Genie Code by reducing LLM judge error by 74-89% on key dimensions. This alignment was achieved with ~50 labeled examples, demonstrating the importance of both semantic and episodic memory for closing the gap between LLM judges and human experts.

From Black Box to Observability: Tracing OpenClaw with MLflow

MLflow Tracing now provides full observability for OpenClaw agents, moving them from black box to transparent. Learn how to quickly set up tracing to understand why your agent makes specific decisions, rather than just seeing the output.

MLflow 3.12.0 introduces multimodal tracing, allowing storage and rich rendering of PDFs, audio, and images as artifact attachments in tracing spans. It also adds AI Gateway guardrails to prevent unsafe model inputs/outputs and extends coding agent tracing support to Codex, Gemini, and Qwen.

MLflow 3.12.0rc0 introduces enhanced AI agent development features, including automatic tracing for more AI coding assistants and OpenClaw, along with new AI Gateway guardrails for safety checks. It also adds multimodal trace attachments for viewing images, audio, and files in the UI, and a new `mlflow.diffusers` flavor for saving and serving diffusion models.

AI observability for production: Seeing Inside Your Multi-Agent System with MLflow

MLflow now offers enhanced AI observability for multi-agent systems, providing crucial visibility into their internal workings. This helps practitioners prevent unintended actions like data purges or sensitive information leaks in production.

Structuring AI Evaluation and Observability with MLflow: From Development to Production

MLflow now offers enhanced tools for structuring AI evaluation and observability, including new APIs and UI features for logging LLM calls, prompts, responses, and metrics. This enables practitioners to systematically track, compare, and analyze model performance and behavior across development and production, facilitating iterative improvement and robust monitoring.

Enforce Content Policies at the Gateway with AI Gateway Guardrails

MLflow AI Gateway now supports configurable guardrails, using LLM judges to block or sanitize harmful content, PII, and custom policy violations. Enforce content policies at the gateway before requests reach your users or models.

TypeScript SDK 0.2.0 RC1

Release candidate for `@mlflow/vercel` TypeScript package with version 0.2.0: https://github.com/mlflow/mlflow/pull/22105

How to Prevent Runaway Agent Costs with MLflow AI Gateway

MLflow AI Gateway now helps prevent runaway agent costs by providing visibility into which part of your agent is driving up costs. This allows you to identify and address cost drivers before investing in the wrong optimizations.

MLflow 3.11.1 introduces AI-powered issue detection for agent traces, budget alerts and limits for AI Gateway spending, and a new interactive graph view for visualizing trace hierarchies. It also enhances security with pickle-free model serialization and improves dependency management with native UV support.

Harness Your OpenHands Agent with AI Observability and Governance

MLflow now supports tracing, evaluating, and governing OpenHands agents, capturing every step of their autonomous operations. This enables practitioners to monitor agent actions, assess output quality, and manage LLM costs effectively.

Testing and Refining Claude Code Skills with MLflow

MLflow tracing and LLM judges can now test Claude Code skills. This enables a self-improvement loop where Claude Code refines its own abilities.

Tracking and Debugging AI Safety Evaluations with Inspect AI and MLflow

Inspect AI evaluations now integrate with MLflow for experiment tracking and execution tracing via the inspect-mlflow package. This enables practitioners to track and debug AI safety evaluations using familiar MLflow tools.

MLflow Workspaces: Shared Deployment Without Separate Servers

MLflow Workspaces are now available, enabling shared MLflow deployments across multiple teams by adding a logical organization and permission layer. This allows teams to scope experiments, models, traces, prompts, AI Gateway resources, and artifacts within their own workspace.

Your Agents Need an AI Platform

MLflow 2.12 ships with new features for building and managing AI agents, including enhanced logging for agent traces, evaluation tools, and versioning capabilities. Leverage MLflow as your unified platform for developing, deploying, and governing reliable AI agents in production.

Control LLM Spend with AI Gateway Budget Alerts and Limits

AI Gateway now supports budget policies to control LLM spend with alerts and request limits. Set spending thresholds, receive webhook alerts, and automatically reject requests when budgets are exceeded.

This release introduces AI-powered issue identification for agent traces, budget alerts and limits for AI Gateway spending, and an interactive graph view for trace hierarchies. It also includes native OpenTelemetry GenAI convention support, Opencode tracing integration, UV package manager support, and pickle-free model serialization options for enhanced security.

This release adds a "try-it" page for Gateway usage examples and filters gateway experiments from the experiment list in the UI. It also fixes numerous UI issues, artifact download problems, and tracing errors, including issues with model copying across workspaces.

Agent Trace Evaluation with TruLens Scorers in MLflow

Evaluate agent traces with TruLens GPA framework through mlflow.genai.evaluate(). Score agent plans, tool calls, and reasoning directly within MLflow.

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

High-level summary: problems, approaches, and takeways for better RAG with MLflow

Ship LLM Agents Faster with Coding Assistants and MLflow Skills

MLflow now provides coding assistants with the required feedback loop to build better LLM agents. Trace, analyze, fix, validate, and repeat to ship LLM agents faster.

Deterministic Safety Checks in MLflow with Guardrails AI

MLflow evaluation pipelines now support fast, deterministic safety validation with Guardrails AI scorers. This enables adding safety checks without requiring an LLM.

Enterprise-Scale MLflow Operations and Security Practices at LY Corporation

How LY Corporation Uses MLflow: An Overview

Deploy MLflow Models to Serverless GPUs with Modal

MLflow models can now be deployed to Modal's serverless GPU infrastructure. This enables auto-scaling and streaming predictions for your MLflow models.

Multi-turn Evaluation & Simulation: Enhancing AI Observability with MLflow for Chatbots

MLflow 3.10 now supports multi-turn evaluation and conversation simulation, enabling scoring of full conversations and reproducible testing of agent changes. This helps catch failures that only emerge across multiple turns, improving chatbot observability.

Introducing MLflow AI Gateway: Governed, Observable Access to LLMs

MLflow AI Gateway provides a single, secure endpoint for all LLM providers, complete with usage tracking and native tracing. This new feature offers governed, observable access to LLMs for Databricks practitioners.

MLflow 3.10.0 introduces multi-workspace support for organizing experiments and models, alongside new GenAI features like multi-turn evaluation, LLM cost tracking, and AI Gateway usage analytics. The UI has been redesigned for improved navigation, and in-UI trace evaluation is now available.

MLflow now supports multi-workspace environments for organizing experiments and resources, alongside a new top-level navigation split for GenAI and Classical ML workflows. Key new features include multi-turn conversation simulation, automatic LLM trace cost tracking, AI Gateway usage analytics, and a CLI command to generate a demo environment.

5 Tips to Get More Out of Your Claude Code with MLflow

MLflow now offers an MCP server, CLIs, and Skills to extend Claude Code, enabling you to trace tokens and monitor tool usage. These five tips will help you transform your Claude coding agent into a transparent and controllable workflow.

v.3.9.0

MLflow 3.9.0 introduces an in-product MLflow Assistant chatbot and a Trace Overview Dashboard for GenAI experiments, enhancing debugging and performance insights. The AI Gateway is revamped for direct tracking server integration, alongside new LLM judge features for online monitoring and custom prompt building.

Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow

Improve your agents using MLflow's extensive, industry-leading suite of high-quality LLM judges.