MLflow

Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code

MemAlign, an open-source MLflow framework, significantly improved the evaluation of traditional machine learning in Genie Code by reducing LLM judge error by 74-89% on key dimensions. This alignment was achieved with ~50 labeled examples, demonstrating the importance of both semantic and episodic memory for closing the gap between LLM judges and human experts.

From Black Box to Observability: Tracing OpenClaw with MLflow

MLflow Tracing now provides full observability for OpenClaw agents, moving them from black box to transparent. Learn how to quickly set up tracing to understand why your agent makes specific decisions, rather than just seeing the output.

MLflow 3.12.0 introduces multimodal tracing, allowing storage and rich rendering of PDFs, audio, and images as artifact attachments in tracing spans. It also adds AI Gateway guardrails to prevent unsafe model inputs/outputs and extends coding agent tracing support to Codex, Gemini, and Qwen.

MLflow 3.12.0rc0 introduces enhanced AI agent development features, including automatic tracing for more AI coding assistants and OpenClaw, along with new AI Gateway Guardrails for safety checks. It also adds multimodal trace attachments for images and audio in the UI, and a new mlflow.diffusers flavor for saving and serving diffusion models.

AI observability for production: Seeing Inside Your Multi-Agent System with MLflow

MLflow now offers enhanced AI observability for multi-agent systems, providing crucial visibility into their internal workings. This helps practitioners prevent unintended actions like data purges or sensitive information leaks in production.

Structuring AI Evaluation and Observability with MLflow: From Development to Production

MLflow now offers enhanced tools for structuring AI evaluation and observability, including new APIs and UI features for logging LLM calls, prompts, responses, and metrics. This enables practitioners to systematically track, compare, and analyze model performance and behavior across development and production, facilitating iterative improvement and robust monitoring.

Enforce Content Policies at the Gateway with AI Gateway Guardrails

MLflow AI Gateway now supports configurable guardrails, using LLM judges to block or sanitize harmful content, PII, and custom policy violations. Enforce content policies at the gateway before requests reach your users or models.

TypeScript SDK 0.2.0 RC1

Release candidate for `@mlflow/vercel` TypeScript package with version 0.2.0: https://github.com/mlflow/mlflow/pull/22105

How to Prevent Runaway Agent Costs with MLflow AI Gateway

MLflow AI Gateway now helps prevent runaway agent costs by providing visibility into which part of your agent is driving up costs. This allows you to identify and address cost drivers before investing in the wrong optimizations.

MLflow 3.11.1 introduces AI-powered issue detection for agent traces, budget alerts and limits for AI Gateway spending, and a new interactive graph view for visualizing trace hierarchies. It also enhances security with pickle-free model serialization and improves dependency management with native UV support.

Harness Your OpenHands Agent with AI Observability and Governance

MLflow now supports tracing, evaluating, and governing OpenHands agents, capturing every step of their autonomous operations. This enables practitioners to monitor agent actions, assess output quality, and manage LLM costs effectively.

Testing and Refining Claude Code Skills with MLflow

MLflow tracing and LLM judges can now test Claude Code skills. This enables a self-improvement loop where Claude Code refines its own abilities.

Tracking and Debugging AI Safety Evaluations with Inspect AI and MLflow

Inspect AI evaluations now integrate with MLflow for experiment tracking and execution tracing via the inspect-mlflow package. This enables practitioners to track and debug AI safety evaluations using familiar MLflow tools.

MLflow Workspaces: Shared Deployment Without Separate Servers

MLflow Workspaces are now available, enabling shared MLflow deployments across multiple teams by adding a logical organization and permission layer. This allows teams to scope experiments, models, traces, prompts, AI Gateway resources, and artifacts within their own workspace.

Your Agents Need an AI Platform

MLflow 2.12 ships with new features for building and managing AI agents, including enhanced logging for agent traces, evaluation tools, and versioning capabilities. Leverage MLflow as your unified platform for developing, deploying, and governing reliable AI agents in production.

Control LLM Spend with AI Gateway Budget Alerts and Limits

AI Gateway now supports budget policies to control LLM spend with alerts and request limits. Set spending thresholds, receive webhook alerts, and automatically reject requests when budgets are exceeded.

This release introduces AI-powered issue identification for agent traces, budget alerts for AI Gateway spending, and a new interactive graph view for visualizing trace hierarchies. It also includes pickle-free model serialization for enhanced security and native OpenTelemetry GenAI convention support for trace export.

This release adds a "try-it" page for Gateway usage examples and filters gateway experiments from the experiment list in the UI. It also fixes numerous UI issues, artifact download problems, and tracing errors, including issues with model copying across workspaces and artifact access permissions.

Agent Trace Evaluation with TruLens Scorers in MLflow

Evaluate agent traces with TruLens GPA framework through mlflow.genai.evaluate(). Score agent plans, tool calls, and reasoning directly within MLflow.

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

High-level summary: problems, approaches, and takeways for better RAG with MLflow

Ship LLM Agents Faster with Coding Assistants and MLflow Skills

MLflow now provides coding assistants with the required feedback loop to build better LLM agents. Trace, analyze, fix, validate, and repeat to ship LLM agents faster.

Deterministic Safety Checks in MLflow with Guardrails AI

MLflow evaluation pipelines now support fast, deterministic safety validation with Guardrails AI scorers. This enables adding safety checks without requiring an LLM.

Enterprise-Scale MLflow Operations and Security Practices at LY Corporation

How LY Corporation Uses MLflow: An Overview

Deploy MLflow Models to Serverless GPUs with Modal

MLflow models can now be deployed to Modal's serverless GPU infrastructure. This enables auto-scaling and streaming predictions for your MLflow models.

Multi-turn Evaluation & Simulation: Enhancing AI Observability with MLflow for Chatbots

MLflow 3.10 now supports multi-turn evaluation and conversation simulation, enabling scoring of full conversations and reproducible testing of agent changes. This helps catch failures that only emerge across multiple turns, improving chatbot observability.

Introducing MLflow AI Gateway: Governed, Observable Access to LLMs

MLflow AI Gateway provides a single, secure endpoint for all LLM providers, complete with usage tracking and native tracing. This new feature offers governed, observable access to LLMs for Databricks practitioners.

MLflow 3.10.0 introduces multi-workspace support for organizing experiments and models, alongside new GenAI features like multi-turn evaluation, LLM cost tracking, and AI Gateway usage analytics. The UI has been redesigned for improved navigation, and in-UI trace evaluation is now supported.

MLflow now supports multi-workspace environments for organizing experiments and resources, alongside a new top-level navigation split for GenAI and Classical ML workflows. Key new features include multi-turn conversation simulation, automatic LLM trace cost tracking, AI Gateway usage analytics, and a CLI command to generate a demo environment.

5 Tips to Get More Out of Your Claude Code with MLflow

MLflow now offers an MCP server, CLIs, and Skills to extend Claude Code, enabling you to trace tokens and monitor tool usage. These five tips will help you transform your Claude coding agent into a transparent and controllable workflow.

v.3.9.0

MLflow 3.9.0 introduces an in-product MLflow Assistant chatbot and a Trace Overview Dashboard for GenAI experiments, enhancing debugging and performance insights. The AI Gateway is revamped for direct tracking server integration, alongside new LLM judge features for online monitoring and custom prompt building.

Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow

Improve your agents using MLflow's extensive, industry-leading suite of high-quality LLM judges.

Introducing MLflow Agents Dashboard

Monitor your Agents with comprehensive analytics and visualizations in MLflow's new Overview tab

MLflow 3.9.0rc0 introduces an in-product AI Assistant for debugging and a new Trace Overview Dashboard for GenAI experiments. The AI Gateway is now integrated into the tracking server, and users can configure LLM judges for online monitoring and build custom judges directly in the UI.

UnityCatalog 0.3.0

Unity Catalog now supports Spark 4.0 and Delta Lake 4.0, enhancing compatibility with the latest Databricks runtime components. New API surfaces for credentials and external locations provide more flexible handling of external storage services.