OpenAI
Recent items mentioning OpenAI across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
OpenAI's newer reasoning models are hallucinating more than prior generations, pushing Databricks to spotlight RAG and evaluation frameworks as mitigations 1. Elsewhere, integration work continues rather than model news: both the Go and Java Databricks SDKs shipped Entra service principal authentication for Azure OpenAI and Microsoft Foundry providers alongside new AWS Bedrock access-key support 78, and Databricks joined the Open Secure AI Alliance with NVIDIA to push AI safety standards industry-wide 5.
Generated daily from the 8 most recent items mentioning OpenAI. Click any [N] to jump to the source.
What are AI Hallucinations?
What are AI hallucinations, why does it matter, and what can enterprises do about it? Newer reasoning models from OpenAI and DeepSeek are actually hallucinating more than their predecessors, not less, making detection and prevention a must for any production deployment. Enterprises can curb the risk with retrieval-augmented generation, domain-specific fine-tuning, systematic evaluation frameworks, and strong data governance.
Managing AI Coding Costs at Scale
--- top comments --- [extr] I would be really curious to hear from devs at Databricks what the experience of development is like internally. I work at a small startup with essentially unlimited AI spend budget - the entire point is that I should be turning to it at every opportunity since our human labor is so expensive relative to tokens. So generally it's like: - Spend most time prioritizing/discussing what to do. - Once that's agreed, use Fable 5 High + 5.6 Sol XHigh come up with a design + plan. Agree on the high level plan. (Usually this just comes down to choosing where the change belongs on the spectrum between minimal patch <-> full redesign) - Use Opus 5 or Sol Med to execute - Auto-fix bugs and CI until green + thermonuclear review skill x3. - Manual interrogation of change/nits - Come up with QA plan and have Codex Computer Use execute on it - Manually spot check the final result (usually a sizable diff, thousands of lines, complete feature E2E, etc) I probably spend like $80 a day at least but I produce the output of 3 or 4 2022 engineers and probably at better quality. So it's easily worth it. Would I save money by switching to GLM 5.2 and such...perhaps? IDK. At our scale it's not worth the time spent building the eval harness to actually understand the performance tradeoff. [lbriner] There are a surprising number of articles like this along the lines of, "we started using AI tools and ended up spending millions per year". On what planet do people start paying for things without keeping an eye on the costs and no-one notices until you have spent a crazy amount? I don't understand. You are either paying a fixed amount which you are happy about in-advance or you are PAYG in which case you would ballpark how much it costs. Otherwise it reads a bit like a fake problem, because it didn't really happen, you just foresaw it (as you should) and added a few guide rails. [sashank_1509] I suspect that when it comes to hard complex software products, you’re better off ignoring agents and doing “trad coding”. What you lose in short term speed you gain in manageable complex codebases. If you have a 500k line codebase and even > 50% is written by agents, you are in a world of pain that won’t justify the costs longer term. Now of course, there are products that just involve lots of code but are not actually complex. This is generally the project with like hundreds or thousands of features but most of the features are separate and don’t actually interact in complex ways. Think a task management app with hundreds of features like calendar, email integration etc. there I think agents gives you more bang for the buck. Just my thought, using agents at work. [platinumrad] Careful. If you admit to using models that weren't trained by OpenAI or Anthropic then you might hauled in front of Congress: https://www.scmp.com/news/china/diplomacy/article/3362616/us... [dgellow] What I take from this is that models are already commoditized, and it’s pretty clear nobody has a moat: routing for the models, they can be swapped whenever new models are released, AI labs will have to continue to run on the treadmill non stop or be replaced. Long term I cannot imagine that business will be high margin. Routing for the harness, so anything that differentiate a provider vs another isn’t exposed to the user and isn’t too relevant. One more datapoint for the thesis that OpenAI and anthropic aren’t viable, sustainable businesses, and cannot justify their $1T valuation and the level of compute commitment (reminder that OpenAI committed to >$750B in infra spending for 2030)
NewsBuilding Agents on Databricks with Custom Apps and Omnigent
This video demonstrates how to build, update, and govern custom AI agents on Databricks using Agent Bricks, Databricks Apps, and Omnigent. The tutorial shows how to integrate Model Context Protocol servers, track execution with MLflow traces, schedule automated agent tasks, and manage security policies through Unity AI Gateway.
TutorialsBuilding Agents on Databricks with Custom Apps and Omnigent
The video demonstrates how to build, update, and govern a store operations AI agent on Databricks using Model Context Protocol servers and custom apps. It shows how to use Omnigent and CodeX to add new context and tools, redeploy the application, and manage governance and traces through the Unity AI gateway.
Databricks joins the Open Secure AI Alliance to advance AI safety and security
Databricks has joined the Open Secure AI Alliance as an inaugural member to advance open AI safety and secure the entire AI stack alongside partners like NVIDIA. To drive this effort, Dat
MLflow 3.15.0 introduces an MCP Registry for registering and sharing Model Context Protocol servers, enhances the Assistant with multi-provider LLM support and per-session token usage tracking, and enables proxy-less artifact transfers via presigned URLs to reduce server load and timeouts on large files. Additional improvements include sharable Runs table views, multi-modal image attachments for LLM judges to evaluate vision tasks, and numerous bug fixes across tracing, evaluation, gateway, and UI components.
TutorialsBuild Your First AI Agent + LLM Tool Calling Explained | Chapter 09
Agents operate through tool-calling loops where the LLM requests specific tool execution with arguments, code executes the tools, and observations return for the LLM to process. The video demonstrates building a Python agent using Grok API with two tools that execute sequentially: retrieving weather for a city and suggesting an activity based on temperature.
The Go SDK adds AwsAccessKey authentication for Amazon Bedrock model providers and EntraServicePrincipal authentication for Azure OpenAI and Microsoft Foundry providers. These new configuration options enable additional credential management methods when integrating external AI models through Databricks.
The SDK adds support for AWS Bedrock access key authentication in the model provider service configuration. New Microsoft Entra service principal authentication fields are now available for Azure OpenAI and Microsoft Foundry provider configurations.
Show HN: Dex – Cost-aware analytics engineering skills for agents
Hi I’m Marco, co-founder of Exmergo. Me and my team created Dex to help Analytics Engineers do real work with Claude Code (and any other coding agent). We’ve found that Data and Analytics teams are stuck between a rock and a hard place cost-wise: - On one side, they are using some of the most expensive consumption-billed software on the planet (Snowflake, Databricks etc.). - On the other side, Anthropic and Open AI want you to tokenmax (and with data analytics it’s very easy to burn your context window). So we created Dex, our open source skills plugin (Apache-2.0), to solve both of these problems: - Dex forces the agent to use a cost guard when performing exploration queries and transformations. - Dex uses a set of tooling that makes it hard to get burned when transforming data. It achieves these things with tight control scripts that the SKILL.md files are pointed towards when using /dex:explore, /dex:transform and /dex:maintain. Bonus: Dex makes agents really good data exploration, building sql and dbt models and detecting drift. 76% performance on ade-bench with Claude Sonnet 5 (and, per our measures, 2.5x cheaper than Fable 5). Install on any agent with this command in your terminal: npx skills add exmergo/dex Install on Claude Code running these commands (separately): /plugin marketplace add exmergo/exmergo-agent-plugins /plugin install dex@exmergo If you want to see more visual examples you can go through the README or browse here: https://www.exmergo.com/dex Let me know if this helps your analytics workflow and makes your agents more cost-aware.
OpenAI and Databricks at DAIS 2026: Making enterprise AI real
Databricks and OpenAI are partnering to make enterprise AI real, combining Databricks' data and AI infrastructure with OpenAI's advanced intelligence. This collaboration helps organizations move from prototypes to production-ready agents, with a joint webinar on August 4-6 to showcase what's next for agentic AI at scale.
MLflow 3.14.0 adds one-command agent setup with Databricks support and durable low-latency Claude Code tracing, Review Queues for trace annotation and feedback collection, and @mlflow.test pytest markers for regression testing. Default model serialization formats change for sklearn to skops, PyTorch to pt2, and LightGBM to skops.
NewsHow LLMs Understand your Prompts: Tokenization & Embeddings | Chapter 05
The video explains how Large Language Models (LLMs) understand text by converting it into numerical representations through tokenization and embeddings. It demonstrates how text is broken into tokens, assigned unique IDs, and then transformed into dense vectors (embeddings) that capture semantic meaning and positional information for LLM processing.
MLflow 3.13.0 introduces Role-Based Access Control with Admin UI, automatic trace archival to S3, and one-click observability for Claude Code and other coding agents. Breaking changes include a redesigned permission system (legacy APIs removed), MLServer removal from pyfunc serving, and requirement for MLFLOW_ALLOW_FILE_STORE=true flag for local file-based stores.
TutorialsHow Large Language Models (LLMs) Work - Full Explanation | Chapter 04
Large Language Models (LLMs) are text-based neural networks trained on massive data to predict the next word (token), operating through tokenization, vector embeddings, and a transformer architecture. LLMs undergo pre-training, supervised fine-tuning, and reinforcement learning from human feedback to become helpful, safe, and aligned, with concepts like context length, knowledge cut-off, and hallucination defining their capabilities and limitations.
MLflow 3.13.0rc0 completely overhauls Role-Based Access Control with unified permission APIs and a new Admin UI, and integrates Claude Code, OpenAI, Ollama, and OpenClaw as native assistant providers in the AI Gateway. The release adds trace archival with seamless retrieval, GenAI agent stress-testing, Kubernetes Helm chart support, and database replica routing for horizontal scaling.
MLflow 3.12.0 adds multimodal tracing with artifact attachments supporting rich rendering, extends coding agent tracing to Codex/Gemini/Qwen platforms, and introduces gateway guardrails for input/output safety. Trace table pagination improves performance, third-party scorer registration enables custom evaluations, and new provider support expands AI Gateway integration options.
MLflow 3.12.0rc0 adds automatic tracing for multiple AI coding assistants (Claude Code, Codex, Qwen Code, Gemini CLI, OpenClaw) through installable TypeScript plugins, and introduces guardrails for production agent safety in the AI Gateway. The release also supports multimodal trace attachments for images, audio, and files with UI visualization, plus a new mlflow.diffusers flavor for diffusion models with LoRA adapters.
OpenAI GPT-5.5 + Codex, now available and fully-governed in Databricks
GPT-5.5 and Codex are now natively available in Databricks, fully governed by Unity AI Gateway for permissions, cost controls, guardrails, and observability. This enables agent building with GPT-5.5 and natural language querying of enterprise data via Genie.
ReleasesHow OpenAI and Databricks are working together
Databricks and OpenAI are partnering to help enterprises deploy and adopt AI, with Databricks focusing on secure data access and management for AI applications through products like Genie and AI Gateway. The video highlights GPT 5.5's enhanced planning capabilities and its leading performance in office knowledge work benchmarks, demonstrating its impact beyond coding to automate internal business processes.
Unity Catalog AI 0.4.0
DatabricksFunctionClient now accepts an optional warehouse_id parameter to execute functions via the Statement Execution API when serverless compute is unavailable, while serverless remains the default execution path. The release requires Python 3.10 or later and fixes issues with Gemini toolkit, LangGraph integration, OSS client function creation, and dependency declarations.
Databricks partners with OpenAI on GPT-5.5
GPT-5.5 and Codex are coming soon to Databricks, governed by Unity AI Gateway, and cut OfficeQA Pro errors nearly in half. This partnership with OpenAI brings advanced models directly to Databricks users.
MLflow 3.11.1 introduces AI-powered issue detection in traces, AI Gateway budget alerts and spending controls, trace graph visualization, native Databricks gateway provider, and pickle-free model serialization. TypeScript SDK packages are now @mlflow-scoped and LiteLLM is no longer required for GenAI evaluation.
ReleasesDatabricks Genie Code, Carl, Bull**** Bench & more! | AI Newsround - March '26 | Advancing Analytics
The video discusses Databricks' new AI tools, Genie Code for autonomous data work and Carl for faster, cost-efficient enterprise knowledge agents using custom reinforcement learning. It also covers the Bench V2 for evaluating AI models' ability to detect and push back on nonsense, along with updates to various models like Qwen 3.5, Gemini 3.1 Flashlight, and OpenAI's GPT-5.3 Instant, 5.4, Mini, and Nano, highlighting their focus on agent capabilities and cost-efficiency.
This release introduces AI-powered issue identification for agent traces, budget alerts and limits for AI Gateway spending, and an interactive graph view for trace hierarchies. It also includes native OpenTelemetry GenAI convention support, Opencode tracing integration, UV package manager support, and pickle-free model serialization options for enhanced security.
NewsOpenClaw, Databricks Agentic Data Monitoring & more! | AI Newsround - February 2026 | Advancing AI
The video discusses OpenClaw, an open-source framework for AI agents, and Databricks' new agentic data quality monitoring solution. It also introduces Advancing Analytics' Lake Forge and Pantheon, a framework and AI layer for developing scalable Lake Flow pipelines, and highlights new model releases from Anthropic, Google, and OpenAI.
Delta Lake 4.1.0
Delta Lake 4.1.0 supports Apache Spark 4.1.0 and introduces conflict-free enablement of Deletion Vectors and Column Mapping on existing tables without blocking concurrent writes. The release requires Java 17 and Spark 4.0.1 or higher (dropping Spark 3.5), adds full catalog-managed table support in Delta Kernel for Unity Catalog integration, and fixes MERGE/INSERT struct expansion bugs.
v.3.9.0
MLflow 3.9.0 introduces an in-product MLflow Assistant chatbot and a Trace Overview Dashboard for GenAI experiments, enhancing debugging and performance insights. The AI Gateway is revamped for direct tracking server integration, alongside new LLM judge features for online monitoring and custom prompt building.
NewsDatabricks: What’s new in October 2025 #databricks news
Databricks introduces Databricks One, a new business-focused experience with consumer access for dashboards and Genie, alongside updates to Genie for defining relations and extended API endpoints. The platform also adds features like easy conversion of external to managed tables, enhanced Databricks Asset Bundles with policy integration and script execution, and new system tables for MLflow tracking and data classification results.
Unity Catalog AI 0.3.0
Functions in Unity Catalog AI 0.3.0 now execute in a sandbox by default rather than the main process, with local and legacy execution modes available. The release adds callable and source retrieval APIs for functions, fixes serverless credential issues, and streamlines toolkit initialization for Databricks compute.
Unity Catalog AI 0.1.0
The initial release of Unity Catalog AI introduces the unitycatalog-ai client package, enabling Databricks practitioners to manage and execute Unity Catalog functions as GenAI tools. This release also delivers dedicated integration packages to seamlessly use these functions within LangChain, LlamaIndex, OpenAI, Anthropic, CrewAI, and AutoGen.
