Anthropic
Recent items mentioning Anthropic across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks Genie now integrates natively with Claude via an MCP server connection and the newly GA Genie Ontology, allowing practitioners to run authorized SQL queries directly within Claude workflows 79. Meanwhile, Databricks introduced Unity Gateway benchmarks evaluating Claude Code's prompt caching and token costs against rival harnesses 6, even as Databricks on AWS disabled access to Claude Opus 5 and Opus 4.8 4.
Generated daily from the 10 most recent items mentioning Anthropic. Click any [N] to jump to the source.
NewsLearn GenAI End to End 💯 #genai #claude #ai #llm
This video teaches data engineers how to upskill in GenAI and build end-to-end RAG chatbots using office data. It follows an experienced professional who overcomes their skills gap by following a structured playlist, building confidence, and leading projects.
TutorialsHow to Reduce AI Agent Costs | Unity Gateway Insights
Unity Gateway Insights helps organizations track AI agent usage and costs. The tool identifies common financial inefficiencies such as failing MCP tools, frontier models performing routine tasks, and repeated work.
What's new in dbt: November 2026
dbt's November release introduces local unit tests, a free dbt State plan, and the public beta launch of Wizard Desktop. The update also brings dbt Charts directly into Claude and ChatGPT.
Claude Opus 5, Opus 4.8 and Fable 5.1 Access Disabled – Databricks on AWS
Subsidized tokens came to Databricks
Now, in Unity Gateway, you can connect a ChatGPT or Claude subscription and use cheaper tokens from there. https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2 submitted by /u/hubert-dudek [link] [comments]
NewsCompare Claude Code, Codex and Hermes on One Model with Unity Gateway
The video compares Claude Code, Codex, and Hermes Agent running on the same model using Databricks Unity Gateway to evaluate performance and cost differences. It demonstrates that while all three harnesses achieve similar correctness on coding tasks, their token usage and cost vary significantly due to differences in prompt handling and caching.
Databricks Genie Ontology is now GA, and new Genie plugins bring it into ChatGPT and Claude. 🧞
Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-genie-dataengineering-ugcPost-7526718327315087360-rXFe/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]
Databricks Doubles Down on AI Products, Opens New 235,000-Square-Foot San Francisco Headquarters
Databricks has opened a new 235,000-square-foot global headquarters at One Sansome in San Francisco, marking a $1 billion-plus investment to expand its AI research and product development. The new facility houses the research and engineering teams building data and AI capabilities for advanced customers such as Anthropic, Cognition, Cursor, Decagon, and OpenAI.
Bring your business context to ChatGPT, Claude, Microsoft Copilot Cowork and Grok Bot
Databricks Genie now integrates natively with ChatGPT, Claude, Microsoft Copilot Cowork, and Grok Bot, alongside an MCP server connection for other AI agents. These external tools can draw on the Genie Ontology to run SQL over authorized data and return cited business answers directly into drafting, planning, or messaging workflows.
Building an AI harness in Genie Code vs Claude Code. Anyone done both?
I run a small consulting firm that builds AI "harnesses" for analytics teams. We connect to the client's data warehouse (in their environment) and handle ad hoc requests, recurring reporting, and the deliverables that go to non-technical stakeholders, like dashboards, PowerPoints, and HTML pages. Usually the harness is a folder structure (SharePoint or a git repo) with context docs, reusable skills, session logs, and a QC steps, and we run it with Claude Code (specifically Claude extension in VS Code). We have a client on Databricks that wants us to build it there instead with Genie Code. So far, it's been better than I expected, but we're still trying to understand if it has any shortcomings compared to our solution. For anyone who has built in both: - Where is Databricks better vs Claude Code (and vice versa)? - What's the easiest way to develop day to day: the VS Code Databricks extension, or just working in the workspace? - I know the setup in Databricks works a bit differently. Is there anything I should know going in, or any recommendations that would make it easier? Anything else? submitted by /u/KungPaoChickenGuy [link] [comments]
Is teaching internal users Databricks still worth it now that AI tools can write the SQL?
We’re a mid-size company with a very small data team, and Databricks is where all our data lives. Most people only see it through dashboards, but more of them want to explore data themselves, and there’s growing interest in things like Genie and AI/BI dashboards. Leadership wants an internal learning program with hands-on workshops to get people comfortable using it. The audience is mixed: a few analysts who know some SQL, but mostly managers and ops staff who’ve never written a query. I’ve been asked to help figure out what goes into it. Before we commit, I’m curious about people’s general experience with this kind of thing. Did it actually change how people used data, or did attendance drop off after a few sessions? And honestly, is it still worth it when tools like Claude or Genie can answer a question in plain English? Should it now be less about SQL and more about asking good questions and checking whether the answer makes sense? Or do people still need the fundamentals to avoid trusting wrong numbers? Would love to hear what worked, what didn’t, and what you’d do differently. Thanks. submitted by /u/ObjectiveAdditional [link] [comments]
NewsDatabricks MemCurrent, Genie Agents, Azure MPC Server & Feature Views | AI Newsround - October 2026
Databricks announced the beta release of MemCurrent for persistent agent memory, along with Genie Flash mode for faster agent responses and general availability of Feature Views. Azure MCP Server 3.0 reached general availability, and several new AI models including Anthropic Claude Haiku 5.5 and Microsoft Decision 1 were introduced.
No data? No problem: dltHub agent builds ingestion, model and metrics so Genie can answer from the data
hey folks, i'm one of the dltHub cofounders (the data engineer one) and we've been working on a meaning-driven pipeline generation workflow. A little context: We created an agent that can talk to Gemini and generate a pipeline from requirements, complete with ingestion, transformation and context (ontology, taxonomy, metrics etc for the transformed data). We basically apply the same concept as when unstructured data gets modeled into a graph, to raw structured data: we look at Pages in the catalog to get the meaning, then we generate a bespoke ingestion pipeline, model it into a canonical data model to work as a virtual knowledge graph, and add ontology, taxonomy, metrics to help a LLM reflect the same meaning as in the catalog Page. Then, everything gets deployed to Databricks - the pipeline, the data, the ontology+taxonomny+metrics where the meaning and data access can live in one place and be governed securely together. What makes this really special? - pipeline, data and meaning are automatically in sync, one source of truth&meaning (no more meaning drift, the system can be interrogated about nuance to) - the auto-modeling process (tech agnostic) creates ai-native gold datasets from raw api unnestings. It uses ontology based data access over canonical model, specifically for collapsing semantics into a high singnal/noise model for LLMs which creates the most accurate retrieval as benchmarked before by other data teams such as anthropic's who also describes canonical datasets data load tool what? in case you are unfamiliar with dlt (data load tool), it is the OSS library for loading data that has everything you need to make data ingestion reliable and easy (99.83% oss observed run success). Our AI transformation harness (not oss but source available and you can try it) includes this canonical modeling bit and a runtime agnostic transformation framework that lets you run your code locally on engines like duckdb the same as in cloud on large engines more info on our blog with video demo submitted by /u/Thinker_Assignment [link] [comments]
Connect Claude, ChatGPT and Cursor to Genie One MCP [CHANGES COMING IN BY NOVEMBER 30]
MLflow 3.18.0
MLflow 3.18.0 adds automatic issue detection, a redesigned evaluation and trace comparison UI, an experimental Skill Registry, and experimental online code-based evaluation that requires a self-hosted server with job containers. Breaking changes include a new search_logged_models sort order, no registered_model_meta in models:/ downloads, and case-sensitive MCP version identity on MySQL and SQL Server, so back up the database and run mlflow db upgrade before upgrading.
Claude Platform now supports native OpenTelemetry observability in Databricks MLflow
Claude Platform's native OpenTelemetry export is now in Public Beta, sending traces from Anthropic SDKs, the Messages API, and Claude Managed Agents into Databricks MLflow, where they're stored in Unity Catalog. Use trace analysis and MLflow Issues to investigate bad answers and spot repeated tool calls, then have domain experts review answers and rerun evaluation on the same test cases to check whether changes improve quality and cut unnecessary work.
Governing Coding-Agent Sprawl with MLflow AI Gateway
The MLflow AI Gateway puts one governed endpoint in front of coding agents like Cursor, Claude Code, Codex CLI, Gemini CLI, and Pi. It routes each request by endpoint name, applies PII and safety guardrails and a budget policy, and traces every call with token counts and agent, provider, and endpoint tags. A runnable demo on MLflow 3.17.1 shows per-provider attribution and blocked bad requests, along with the latency cost of LLM-judge guardrails.
ACTUALLY USEFUL Big Data Project Ideas
Heyy everyone! We have a Big Data class project proposal coming up, and I want to make an actually useful project (something challenging, or something that could sell, or something that solves an actual problem people have). Not yet another fancy showpiece class project. Requirements: The dataset must be at least 10GB, be genuinely large-scale and require distributed processing The project must include some or all (or even more): data warehouse (snowflake/redshift/any other) airflow elt/dbt spark, sparkml mlflow kafka/spark structured streaming/flink nosql, vectordb llmops Practitioners, help me with some ideas? (ChatGPT and Claude didn't help much). submitted by /u/mulberry-cream [link] [comments]
I Built a Claude Dashboard on Databricks Genie, Then Rebuilt It in AI/BI: The Honest Comparison
NewsAutomatically Route Every Task to the Right Model with Unity Gateway
Unity Gateway manages multiple AI models through a single governed platform and uses smart routing to automatically direct tasks to the most cost-effective and capable model. The demonstration shows how to configure model services, traffic splits, and fallbacks while tracking usage across different development tools.
Release: v2.23.0 (#2277)
The extension now has a databricks.notebooks.configureJupyterCellMarkers setting to stop it writing Jupyter cell-marker settings to workspace settings, and claude and codex are available in SSH tunnel connections with Unity Gateway preconfigured. The bundled Databricks CLI moves to v1.23.0, which restores legacy-import uploads for bundle deploys (so notebooks are no longer uploaded as plain files) and adds workspace.sync_mode, replacing the deprecated presets.source_linked_deployment and experimental.immutable_folder.
Databricks CLI v1.23.0 makes databricks ssh connect provide claude and codex on dedicated clusters and in serverless IDE sessions and non-interactive commands, and adds GPU_1xRTX6000 workload support plus workspace.sync_mode (overwrite, in-place, immutable), which replaces the deprecated presets.source_linked_deployment and experimental.immutable_folder settings. The release also reverts bundle file uploads to the legacy /workspace-files/import-file endpoint to fix notebooks being uploaded as plain files, adds lifecycle.depends_on and bundle deploy --select for deletions, and fixes bundle validate errors when a schedule, trigger, or continuous block is referenced as a whole.
How KOHO built a Claude-powered self-serve analytics tool on years of dbt governance
KOHO built a Claude-powered self-serve analytics tool on top of years of dbt governance, and argues that AI analytics only works when a governed dbt foundation and a strong data culture are already in place. The post walks through why that groundwork, rather than the model itself, is what makes self-serve analytics trustworthy.
How KOHO built a Claude-powered self-serve analytics tool on years of dbt governance
KOHO built a Claude-powered self-serve analytics tool on top of years of dbt governance work, and the post argues that this foundation is what makes AI analytics viable. The takeaway for Databricks teams: a strong data culture and a governed dbt layer need to come before AI self-serve, not after.
Unify your productivity and AI data with Lakeflow Connect
Lakeflow Connect now offers native, fully managed connectors for Atlassian (Jira, Confluence), Microsoft SharePoint, Google Workspace (Drive, Outlook, Gmail), GitHub, and AI providers like OpenAI and Anthropic, with unified governance and no infrastructure to manage. Because both structured and unstructured data land directly in Unity Catalog, teams can power enterprise knowledge bases, AI agents, and developer productivity dashboards without building custom API integrations or pipelines.
What Okta learned governing context for AI agents at scale
Okta moved its AI agent context out of the Snowflake UI and into dbt, using a three-layer framework of agent metadata in model YAML, markdown context docs, and semantic views in dbt's semantic layer spec, all shipped through normal pull requests. The context syncs into Agents Schema, so Claude (over MCP) and Snowflake Cortex read the same governed rules, and an analytics engineer fixes a wrong answer by merging one change that every agent inherits.
How DocuSign cut analytics engineering time in half by pairing Claude Code with dbt v2, dbt MCP, and agent skills
DocuSign combined Claude Code, dbt Agent Skills, dbt MCP, and dbt v2 into a single analytics engineering workflow and measured a 50% productivity gain. The post walks through how these pieces fit together so Databricks practitioners can see what an AI-assisted dbt workflow looks like in practice.
TutorialsBuilding an Agentic Analytics Foundation on the Lakehouse
The video demonstrates how to build an agentic analytics foundation on the Databricks lakehouse platform to ensure AI agents are accurate, secure, and scalable. It explains how to organize data assets into business domains, define metric views, and use Unity Catalog and Unity Gateway for governance and guardrails.
Databricks genie code changes and dbt
Hi I created my first pipeline in Databricks that sits as a .sql file. I've broken up the code into layers (stg, marts) and its working as intended. However, when the code needs to be adjusted (e.g. business definition or logic changes), I often use the genie (it's the databricks in house agent) to help me make the changes. The problem I'm running into is that since those changes are made inside databricks, the dbt files are not adjusted. I currently update the code in databricks then paste the new adjusted code into dbt but this seems backwards to me (Also a challenge when the change ripples throughout the code so I would need to update each individual stg). I imagine the changes should initially happen in dbt then that flows into databricks. We don't have claude that can link with dbt (we're only using genie). Is there a way to link genie onto dbt or is that a limitation? Any best practices in this process I should consider implementing? submitted by /u/Takoyaki_18 [link] [comments]
🚀 Massive updates came to Databricks Unity Gateway this past September!
Enterprise teams gained access to an expanded, multi-provider model hub managed directly through Foundation Model APIs . Whether you needed cutting-edge enterprise reasoning, lightning-fast inference or lightweight task processing, Unity Gateway expanded its lineup with the latest releases from OpenAI, Anthropic, xAI and DeepSeek. Here is a recap of the foundation models that landed on Unity Gateway throughout September 2026: 🤖 OpenAI GPT-6.1 Sol GPT-6 Sol & GPT-6 Luna GPT-6 Astra ⚡ xAI & DeepSeek xAI Grok 4.7 DeepSeek V4.1 Flash 🧠 Anthropic Claude Opus 5.5 Claude Fable 5.1 submitted by /u/Youssef_Mrini [link] [comments]
5 Simple Tips to Supercharge Your Claude SDK Experience with MLflow
Claude SDKs now emit OpenTelemetry traces natively, allowing you to seamlessly monitor and evaluate your agent runs with MLflow. Learn five practical tips to capture unified end-to-end traces, track token usage, score executions, and compare performance across different prompts.
Databricks Asset Bundles now utilize a multipart upload endpoint for improved deployment throughput and automatically retry operations encountering transient server errors. This release also adds uv workspace support for local environment setup alongside fixes for bundle deployment binding, immutable folders, and relative path resolution.
How to trace Claude agents via Native OpenTelemetry Export
Claude now natively emits OpenTelemetry traces across Claude Managed Agents, the Messages API, the Claude SDKs, and Claude Code. Learn how to configure the export for each product using open-source MLflow as your trace backend.
MLflow 3.17.1
MLflow 3.17.1 introduces the mlflow.genai.databricks.review_queues module for managing review queues on Databricks and adds Decision API support to make_judge(). It also fixes Claude Code OpenTelemetry trace rendering in the UI and ensures Databricks evaluation feedback correctly records the latest scorer version.
Letting devs use any AI coding agent against governed data, without keys on laptops. What actually works!!!
Disclosure up front: I'm a forward-deployed engineer at Databricks, so I work on this, but this is a hands-on write-up, not a pitch, and the pattern's the interesting part. The problem I keep hitting at enterprises: devs already use a mix of AI coding agents (Claude Code, Codex, Cursor, Copilot). Leadership wants to "standardize on one," which fights reality. The real question isn't which agent, it's how to let all of them touch enterprise data safely. I spent a week wiring agents through a governed gateway + MCP and trying to break it. What held up: - per-user/per-model entitlement enforced server-side (not a client checkbox) - governed data tools that are fail-closed. no valid identity, no access - reads scoped to the user's catalog permissions. the agent sees only what the human can - human-in-the-loop by default (shows the query before running, no silent writes) Write-up: https://wesdias.medium.com/how-to-connect-any-ai-assistant-to-governed-enterprise-data-with-mcp-0d9cffa14231 How are you handling this? governing agent data access at the gateway/MCP layer, or still per-tool API keys and hope? What's fallen over for you? submitted by /u/wesleydias84 [link] [comments]
NewsWhy Is Everyone Using the Databricks AI Platform Kit?
The Databricks AI platform kit enables users to quickly configure a new workspace through a terminal installation process. The tool generates and executes Terraform scripts to automatically deploy necessary cloud resources, networking, and Unity Catalog structures.
ReleasesJev, OpenAI Dots, DevDay, and Genie One! | AI Newsround - September 2026
The video covers the rise of Jev and other system one decision-making models that output typed probabilities instead of free text for enterprise workflows. It also highlights OpenAI DevDay announcements including persistent agents called dots, the general availability of the Genie one Model Context Protocol, and several new model releases from Anthropic and OpenAI.
Genie Reading Jira Tickets
Has anyone figured out how to get genie to read Jira tickets? I point cursor/claude code at Jira and they fire away. However sometimes I’d like to develop with Genie. Is this possible? submitted by /u/Known-Delay7227 [link] [comments]
Lakebase and Agentic SDLC: Branching Databases for Coding Agents
Lakebase resolves the database bottleneck for parallel coding agents by providing sub-second, scale-to-zero copy-on-write database branching for each agent. Learn how to implement an end-to-end workflow pairing Claude Code, Git worktrees, and GitHub Actions to run Drizzle migrations, deploy preview environments on Databricks Apps, and test against Unity Catalog-masked data.
MLflow 3.17.0
MLflow 3.17.0 introduces fine-grained access control for runs and traces, links evaluation metrics directly to managed datasets, and adds support for Databricks Unity Catalog model service resources. Self-hosted SQL tracking servers require a coordinated schema migration via mlflow db upgrade with all writers stopped, as rolling deployments are not supported.
Configure Unity Gateway rate limits for system.ai model services
I am trying to configure rate limits for system-provided model services under system.ai where I am unable to set the rate limit for the model. Below is the scenario I tried When I open Unity Gateway, select system.ai.claude-sonnet-5, and try to configure the rate limits, the UI displays: "User does not have permission to access this resource." I see the same behavior for other system.ai models. I have Account Admin access at the Databricks account level. However, when I run the following SQL command: SHOW GRANTS ON SCHEMA system.ai; the result only shows the following permissions for account users: EXECUTE READ VOLUME SELECT USE SCHEMA There is no MANAGE privilege shown on system.ai. Additional observation When I directly call system.ai.claude-sonnet-5, I receive HTTP 403 with the following message: PERMISSION_DENIED: The endpoint is temporarily disabled due to a Databricks-set rate limit of 0. I am not sure whether this is related to the inability to configure the rate limit in the UI or whether it is a separate platform-side restriction. Could someone please help me to understand the distinction between account-level administration and Unity Catalog / metastore-level permissions for system.ai model services, and understand the correct least-privilege configuration needed to manage Unity Gateway rate limits?
What changed on Lakenaut after your feedback, and the site is now an MCP server you can plug into Claude
First of all, I really want to thank you everyone commented my latest post and encourage me to keep update and mantain the website. For the ones who miss my post is this one (for reference): https://www.reddit.com/r/databricks/s/S06gTMmm2P A few of you wrote in over the last two weeks, and most of it is live now. The short version, then one bigger thing. From the feedback "Too many categories, the docs' hierarchy is clearer." The sidebar groups now follow the Databricks documentation's domains (Platform & governance, Data engineering · Lakeflow, Data warehousing & BI, AI & ML), while each area keeps the name the workspace menu gives it. The sidebar is contextual: the doors are always there, the tree opens only where you are. "Finishing a page and moving on is one gesture." It is: Mark as done and Done, next sit together at the foot of every concept. "I want to know what moved." Every concept, area, path and roadmap now opens with a "What changed, and when" box, and /changes/ is the whole log by day: which page, what was done, and whether a person did it by hand or accepted a correction the agent drafted after the documentation changed. Every end-of-support row has its own address now, and search lands on it instead of on the list. A fifth way through the material is taking shape: practices, decision pages filed on the well-architected pillars. When to pick a thing, for whom, what choosing wrong costs. The first one is in draft. The bigger thing: an MCP server, no key Put the MCP URL (page below) into Claude, ChatGPT, Cursor or Claude Code and the assistant reads the published site instead of guessing: 170+ concepts, 8 paths, 5 roadmaps, the news, the resources, and the platform tables (what is in preview, what was renamed, what is ending). Thirteen read-only tools, 60 calls a minute per address, every answer links back to the page it came from. How to connect it: https://lakenaut.dev/developers/ Two questions If you plug it into an assistant, what is the first thing you ask it? I want to know whether people reach for prep, for "what is in preview", or for something I have not thought of. Is there a kind of page you keep wishing existed? The practices came from one comment. The next kind could come from yours. Again, thank you and let's connect! submitted by /u/imthenitto [link] [comments]
Genie Code vs. Omnigent: How to structure workspace instructions, AGENTS.md, and skills without breaking production
Following up on recent discussions around Genie performance I've come across... a lot of us have a similar frustration with Genie. Context bloat, prompt drift, and agents making destructive errors (Junior mistakes like setting a mode to overwrite). Part of this is limitations of genie, some is instruction "stuffing" and some is using the right tool for the right job. Also, guidance on how to structure and create instructions have changed a lot over the summer. With Omnigent being available alongside Genie Code, here is my opinion on what should separate these tools and how to structure context files to keep agent outputs deterministic. I made a table because post sprawl is real... Genie Code vs. Omnigent These are not competing options. They solve problems in different places and scales: Layer Genie Code Omnigent (Meta-Harness) Primary Role Native in-workspace assistant for notebooks, SQL Editor, and Lakeflow Pipelines. Unified meta-harness to run, swap, and persist multi-agent workflows (Claude Code, Codex, Cursor, PI, Hermes) across local runners and cloud sandboxes. (Also, try out Debbie that has the ability to use a /debate skil) Governance Prompt-level instructions, workspace instruction files, and Unity Catalog user permissions. External, stateful policies written in CEL (hard cost limits, network sandboxing, pre-commit git checks). Best For Interactive notebook debugging, quick PySpark logic, end-to-end Lakeflow pipeline scaffolding. (Think StackOverflow, for those of us of that still remember it) Multi-repo refactoring, team co-driving sessions (eg. "Multi-player" agent sessions), and running autonomous CLI agents with guardrails. Execution Model Synchronous, interactive execution tied to your active workspace session and compute. Asynchronous, persistent execution that runs in the background (survives laptop close) with shareable web/desktop sessions. My Rule of thumb: If you are working interactively / ad-hoc in Databricks and need help with syntax, where something is, why something is failing, use Genie Code. If you are doing anything complex, want to be able to get one harness to check another one (like Claude Code checking Codex's response/PR) and need programmatic guardrails that can't be bypassed by prompt injection, go use Omnigent. And if you are using Genie and want it to read your "instructions": Databricks does not natively parse an arbitrary instructions.md dropped in a folder. If you want repo/project-level discovery, the standard Genie Code looks for is AGENTS.md (or CLAUDE.md ). Shoving context for "multiple project islands" into one file guarantees prompt dilution. Workspace defaults ( .assistant_workspace_instructions.md ) belong in under 30 lines. Scoped project rules belong in AGENTS.md . Task-specific behaviors belong in modular skills under .assistant/skills/ called via @skill-name . Andrej Karpathy created a git repo "karpathy-guidelines" that walk through some of the overarching concepts he recommends around creating skills and instructions. Specifically one of the ones that may come as a surprise is that if you repeat a instruction point it will lower the effectiveness of the instructions and cause degradation. submitted by /u/OkImprovement7010 [link] [comments]
Genie is so Dumb and I am tired of Pretending Otherwise
Hey Guys, Data Engineer with 3.2 YOE here, I have been working on and off on databricks for multiple projects across domains like Retail, logistics and Now Pharma. In my current role, I focus on creating end to end pipeline with different data sources which often get refreshed on a bi-weekly basis for pharma related data The client is extremely demanding and has no technical expertise to gauge how much time it really takes to build and maintain something so complex and therefore expect the team to use Claude teams license and also Genie The problem with Genie is that, I can not trust it, It will not listen to my instructions even after having a separate instructions.md which I update on a daily basis, in fact I update this after each session is complete, and no, I do not use AI to update this, every change request that the client and the client's team has goes through my own words of instruction updates, I have an entire markdown file which has contexts for multiple islands of projects, workspace folders and notebooks that I have to maintain and transition to devops team. I also have created 4 different skill family for Genie, in relation to migration, devops, maintainence and review. And it is so disgustingly bad at handling all this context, that it fails me across all 4 areas. I have seen it lie to my face multiple times, Casting columns as null, altering data types without asking consent, never following my work stream and process in a sequential format Having set the wrong notebook paths in a task even though i explicitly tell it exactly what to do. It often confuses streams of works and ends up mixing so many things that remove this knot and fuck up itself costs me so much time!!! It is very poor in adhering to the domain specific instructions that we provide More than 3 requests in a chat, and it is practically useless And branching of chats is the most useless feature that they have introduced. I have created multiple diagnosis queries to check the work of this agent , and it will straight up lie to you, so much, so convincingly, that you know it knows all the biases you have and it even goes a step ahead and just narrates a story that you can provide to your team in the stand-up. My only concern is, after all this bullshit, my team, of nearly 6 (some of them have never worked in databricks or delta table environments like this) was charged $990 dollars in the month of august , for this shit output??? Have some shame Databricks , fix your worthless product or remove that feature or stop charging so much if you are beta testing in actual prod. Today, I am writing this post because of something that I caught live, that pushed me to the edge. Something that is critical, production level issue, which If I was not paying attention while merging could have been a huge disaster, and mind you the pipelines I make are client facing, what is even more distrubing is the fact that it will just make so many unnecessary changes to a simple query or a pyspark function just enough so that the tests are passed. (So basically it does not want to get caught, and makes a mistake so that we can prompt it again to fix this mistake and then Databricks can charge us more, this looks like a dark pattern to me) In your preview (prima-facia), everything is good, the tests are passing, the job is running, but holy!!!, it was casting 3 columns which are essential for downstream processes as null. It boldly suggests that we remove those 3 columns, and or cast them as null. How is this a solution Databricks? The AI is supposed to have more context than me because it is a agent made specifically for Databricks Environment Correct?? That is how it is sold?? And upon spending just 2 mins, I realized that the fix that it is suggesting will blow up critical information that the client team should see in the app because it is not even there in the first place, and all of this is because it can not read the correct notebooks , even if you tag it. It had se […truncated]
I am trying to build an interactive dashboard on the underlying Databricks. Which of these are the best?
I’m thinking of 4 options here 1. Build an MCP (a custom MCP) that can access custom tools on Databricks and interface it on Claude.ai or Claude desktop Advantage- Claude is very good at inferencing, multi-turn conversation and multi-step processing Disadvantage - custom MCP and tools needs to be built accurately and validated. It should have full context of schema and unity catalog Build a semi-custom MCP - this will use “askGenie “ as one of its tools with additional custom tools Advantage- complexity decreases as we leverage genie space Disadvantage- double inference by genie and Claude Use custom Databricks connector in Claude. Not sure if this uses genie and therefore double inference but it’s more reliable than custom build because this is a native offering by vendor Use only genie and build custom dashboard without needing Claude interface What’s the thought on this? submitted by /u/Bala_Devaraj [link] [comments]
NewsParallel Coding Agents with Lakebase | Claude Code + GitHub Actions
This video demonstrates how to run multiple coding agents in parallel by combining git worktrees, GitHub Actions, and Lakebase database branching. It shows how each agent automatically receives an isolated database branch for safe experimentation and schema migrations, followed by dedicated preview environments for pull requests.
The CLI introduces a preview image push command for the Databricks Artifact Registry, preconfigures coding assistants in serverless SSH sessions, and adds expiration flags for Postgres branch creation. In Asset Bundles, clusters now support library configurations, Terraform deployments migrate to the direct engine prior to execution, and Postgres snapshot schedules are restricted to YAML.
Azure Databricks, two-week-old account: partner pay-per-token endpoints (Claude, Grok) have never once succeeded — "Databricks-set rate limit of 0" — and the Assistant has "no daily token allowance". Open models work. What gates this?
Azure Databricks, Premium, one account with three workspaces, Unity Catalog, Azure Marketplace billing (no card, no trial credits left). The account was bootstrapped on a trial workspace about two weeks ago and moved to Premium six days later. I've spent two days on this and want a sanity check from anyone who has seen it. The facts, from system.serving.endpoint_usage: • Every open-weight pay-per-token endpoint has worked since the trial: gpt-oss-120b has ~50 successful calls going back to the trial period, Llama 3.3 70B likewise, zero refusals. • No partner pay-per-token endpoint has ever returned a success on this account. The very first call anyone made to databricks-claude-sonnet-5 was refused, and every call since, on every Claude endpoint and on Grok, in all three workspaces, for users and service principals alike: 403 PERMISSION_DENIED: The endpoint is temporarily disabled due to a Databricks-set rate limit of 0. • It is not our AI Gateway config: I removed every rate limit from one Claude endpoint and invoked it — same 403. The endpoints' config shows nothing else. • The notebook Assistant worked during the trial. On the paid account it refuses everyone, admins included: *"This workspace has no daily token allowance for the assistant."* Underneath, /ajax-api/2.0/conversation/llmproxy/ returns 429 {"type":"daily_token_limit_reached","daily_limit_tokens":0,"message":"…Daily token limit of 0 reached. Resets at midnight UTC. (DTB)"} and it recomputes to 0 every midnight. • Once, the Databricks Apps in the workspace were stopped by the platform with *"App compute was stopped due to workspace or account status"* while every workspace showed RUNNING. apps start brought them back. Red herring, for completeness: we also created a Unity AI Gateway per-user budget whose first version had a $0 threshold with BLOCK_USAGE. That blocked Genie for a day, exactly as documented, and was fixed. It was created a day *after* the first Claude refusal, so it isn't the cause of any of the above. What I've found: two Community threads describe "Databricks-set rate limit of 0" as a workspace trust-tier gate — trial-born accounts sit in TRIAL_VERIFIED, pay-per-token partner models are gated to PAYABLE_VERIFIED, a card alone doesn't flip it, only Databricks Sales/Support moves the tier. Both were AWS/personal accounts, neither resolved on-page. The account console's own API reports our account's feature_tier as STANDARD_W_SEC_TIER although the workspaces are Premium; I can't find what that field means. Questions • Does Azure Databricks with Marketplace billing go through the same PAYABLE_VERIFIED gate? Does it clear on its own after the first settled invoice, or does someone have to move it? • If you were moved: who did you contact (account team, or "Contact us" in the console) and how long did it take? • Is the Assistant's "daily token allowance" the same gate, or a trial allowance that simply goes to 0 when the trial ends on an unverified account? • Does feature_tier: STANDARD_W_SEC_TIER on the account object mean anything to anyone? submitted by /u/sumit671 [link] [comments]
Let's talk about AI rate limits
Databricks Unity Gateway is great and getting even better very quickly. It now has nearly everything we could hope for. New frontier models are usually available within 24 hours of release. But...the models aren't really available . They all come with extremely low rate limits, way too low to support a team of engineers in any sizable enterprise using the models. If you have a sizable agents.md even 1 person hits the limits. Why is it like this? Sure we can raise a ticket to request higher limits but that takes weeks and the highest we've gotten is 1M tok/min. We can (and do) go to Bedrock and get 5M+ tok/min by default, no special request needed. Is the expectation that we hook up other providers? Are the key frontier labs (OpenAI, Anthropic, Google) limiting Databricks as a whole? For such a great product this seems like a glaring limitation that's holding us back. Make it make sense. submitted by /u/degenbets [link] [comments]
Claude Academy's Using Databricks for Data Analysis - This official documentation from Anthropic on Claude Academy for how to work with Databricks is really underwhelming, so I wanted to share a guide that properly explains everything.
Hey Databricks community! I have been working with Databricks as an admin for over 4 years and I was attempting to follow this official tutorial from Anthropic for how to connect Claude with Databricks https://academy.claude.com/tutorials/using-databricks-for-data-analysis And I encountered multiple significant frustrations with doing it, as the tutorial is very underexplained and outdated for how to get Claude working with Genie One and other functions within Databricks. Therefore, I thought this video on Youtube would be valuable content to share to help solve this problem, for other people that are encountering similar difficulties when getting these two tools connected and building out systems that work for Claude to properly interact with Databricks. Let me know if this video is helpful! submitted by /u/k_kool_ruler [link] [comments]
ReleasesThe Unity Gateway Cli
The Unity Gateway CLI allows users to configure and manage governed AI coding agents directly from the terminal with workspace authentication and budget tracking. Administrators can centrally configure smart routing, model selections, MCP servers, and skills via JSON to distribute standardized AI tools across development teams.
Deploy and manage coding agents at scale with the Unity Gateway CLI
The new Unity Gateway CLI enables organizations to centrally deploy, configure, and govern coding agents and frontier models at scale. Developers can launch approved agents with a single command, while admins manage models, shared MCP tools, spending policies, and lakehouse trace exports from a unified interface.
Databricks Cost Optimisation 3rd Party Tools
Does anyone have experience with third party optimisation tools like espresso.ai or Zipher? Espresso has an enticing pricing model but I wonder if dumping a whole heap of logs and system stats into Claude would get me 80% of the way there. submitted by /u/Agitated-Western1788 [link] [comments]
Genie One MCP is now GA
The Genie One MCP server is now GA. It exposes Genie One over MCP, allowing any agent to communicate with Genie One as a peer agent. The MCP exposes tools for asking questions to Genie One, getting query results, checking on incremental progress, and steering responses. These capabilities allow you to integrate Genie One into whatever agent you want without changing your workflow. And you get all of the semantics built-in instead of having to use raw SQL APIs with fragmented skills/stale markdown repos/additional semantic layers like we used to with ai devtools. Here are the docs: https://docs.databricks.com/aws/en/agents/mcp/genie-mcp I've seen some pretty cool use cases with integration into ChatGPT/Claude, but also headless agent workflows where you want to delegate the data questions to Genie One so it can use agents/ontology. Interested in folks' thoughts on the right/wrong use cases for this feature. submitted by /u/lakehouse_vacation [link] [comments]
Can Jev replace your LLM judge? Evaluating quality, cost, and latency
Benchmarking Jev against GPT, Claude, and DeepSeek using MLflow reveals how it measures up as an alternative LLM judge. Review side-by-side performance across quality, cost, and latency to see if Jev can replace your current evaluation models.
The Web Search Your Agent Inherited Isn't Good Enough
Omnigent lets you define an agent once—model, tools, policies, limits—and deploy it across Claude Code, Codex, or the raw API instead of rebuilding it per harness, with model calls routed through Databricks Foundation Model APIs for unified cost tracking and governance. Swapping in Nimble for web search lifted benchmark accuracy from 46% to 71% while cutting search costs in half.
NewsAI Watermarking, Databricks AI Extract, Hugging Face, and Unity Catalog | AI Newsround - Summer 2026
Claude and other AI models will invisibly watermark generated text to ensure content traceability, while Databricks AI Extract achieves 94.7% accuracy on complex documents and extracts charts as structured data. Databricks added skills to Unity Catalog as governed assets and launched AI Gateway to centralize access to multiple AI models, enabling rapid evaluation and swapping without code changes.
Show HN: Like LeetCode, but multi-file and multi-step
On LeetCode you write one function, it passes, and you're done. Most AI labs such as Anthropic and OpenAI, as well as some big tech companies such as Meta, Snowflake and Databricks, don't stop in one function and instead ask multi-step questions that are closer to real-world problems. Say you're building a key-value store. Step 1 is get, set, delete, count. Once your tests pass, Step 2 unlocks: expiring keys. Then nested transactions. Then atomic batches. I built this because I couldn't find a good place to practice for multi-step, multi-file questions. The questions are based on ones people have reported from those loops. Check it out at https://loopprep.dev. There are 24 questions, 3 free, and no signup to try them. Happy to answer questions. --- top comments --- [MiroslavPokorny] Writing code in a browser like this or leet code is wrong on so many levels its not funny.
Where Omnigent Fits Alongside ChatGPT Work, Claude Cowork and Cursor Projects
datatf: Automate importing Databricks workspace into Terraform
DataTf , bring an existing Databricks Workspace into Terraform (IaC). generates dynamic terraform.tfvars + import code built on the Databricks Go SDK compatible with Terraform or OpenTofu compatible with Databricks Omnigent, Claude Code, opencode, OpenAI Codex and others native support for Azure, with GCP and AWS support upcoming Disclosure: I am the Author/Founder, 536 Technologies. submitted by /u/536tech [link] [comments]
Get Tuesday's version of this
Tracking Anthropic? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.