AI Agents

Knowledge bases in medallion architecture

Would you put knowledge bases in the bronze/silver/gold layer? The raw documents definitely reside in the bronze layer. But if I create AI Agents atop a volume storing the raw documents, then the knowledge base remains in bronze. However, if I create vector embeddings/do chunking/create a vector search index, then these tables should be in the silver layer. Am I on the right track?

Agentic Data Engineering with Genie Code and Lakeflow

The next generation of Databricks Genie just launched. Here is what data engineers actually need to know.

I have been following Genie since it first launched with AI/BI last year. Back then, I honestly thought it was mostly for business users. A chatbot on top of your data that could answer basic questions in plain English. Useful, but not something I thought data engineers really needed to care much about. After seeing the new 2026 version, I completely changed my mind. Genie is no longer just a business chatbot. The biggest change is Genie Code, which is basically an AI agent designed for data professionals. It can generate pipelines, debug failures, create dashboards, monitor systems, and work directly with Lakeflow and Unity Catalog. That part caught my attention immediately because it moves beyond simple Q&A and starts touching actual engineering workflows. What surprised me most is how connected the whole system has become. It can pull context from dashboards, Genie Spaces, apps, metadata, documentation, and external systems like GitHub, Jira, and Confluence through MCP. Instead of only searching tables, it tries to understand relationships across the environment. That feels very different from the first version. The operational side is also interesting. Genie Code can monitor pipelines, investigate failures, help with DBR upgrades, and respond to issues before teams even notice them. The more I read about it, the more it felt less like a chatbot and more like an assistant sitting beside the engineering team. But honestly, the biggest takeaway for me is not the AI itself. It is what this means for data engineers. A lot of people immediately jump to “AI will replace data engineers,” but I think the opposite is happening. These systems are only as good as the data foundation underneath them. If metadata is incomplete, if tables are messy, if naming conventions are inconsistent, or if documentation is missing, the AI layer will give poor answers confidently. That means clean data modeling, governance, metadata, documentation, and data quality are becoming even more important than before. The engineers building those foundations become more valuable, not less. I think the role is slowly shifting away from spending hours writing repetitive boilerplate transformations and more toward building trustworthy, AI-ready data systems. One thing I keep noticing while learning Databricks through BricksNotes and the wider community is that the platform is moving very quickly toward AI-native data engineering. Features like Unity Catalog, Lakeflow, and now Genie all connect together. It feels like understanding metadata and governance is becoming just as important as understanding Spark itself. Also interesting that Genie now has a full mobile experience on iOS and Android. Business users can access dashboards, apps, and chat directly from their phones, which means the underlying data quality matters even more because people are going to depend on these systems everywhere, not only during work hours. Curious if anyone here is already using Genie or Genie Code in production. I would genuinely like to hear how the answer quality has been and whether your teams are changing how they approach metadata and documentation because of it.

Show HN: Recursant – service mesh for governing AI agents

Hello, I have just released Recursant to the public. I have been working on it for a while. It is a control plane for governing AI agents across stacks. It provides full observability, guardrails, and control on the network level by routing all traffic through a side car. Problem statement: many large, regulated enterprises (think banks, telcos) have one engineering team on LangGraph, another on CrewAI, marketing on AgentForce, and data teams on Databricks Agent Bricks. They need their agents to talk to each other with consistent policy enforcement, one audit trail, and a single set of guardrails, yet allowing different functions to run on their own stacks. Recursant solves that problem using the service mesh pattern Recursant has two components: a registry and the mesh . The registry contains all live agents. The mesh uses sidecars to route traffic and enforce on the network layer. Aim is for Recursant to provide a real-time EU AI Act Annex IV compliance, so it is not generated from static documents. This saves time and effort for large enterprises subject to the requirement. Linmitations: - Recursant currently plugs in to CrewAI, Langgraph, and n8n . The aim is to support proprietary platforms such as ServiceNow and AgentForce as much as psosible. - The Recursant SDK still needs work to support as many agents as possible - I would also like to provide support for some of the 'personal agent' platforms such as OpenClaw, NanoClaw, and Hermes - Only tested on k8s, not public cloud - Documentation is sparse and needs to be developed. I hope this project is useful to some of you. --- top comments --- [goodra7174] clawdlinux.org building the Kubernetes-native runtime that AI agents call to provision their own secure execution environments — the missing infrastructure layer for every enterprise that can’t send agent data to the cloud #Ycombinator #buildinpublic #startups

Claude code + AI Dev kit vs Genie code

Hi everyone, i am testing the limits for agentic engineering for databricks. Has anyone tried using claude code with AI dev kit and comparing it against Genie code? Specifically around these tasks 1) Data engineering 2) ML prototyping 3) AI/BI Dashboard building So far, claude code + AI dev kit seems to work really well. But from a conversation with claude it seems genie code out of the box is better already. Anyone have any experience?

Possible degradation of Genie agentic capabilities in Databricks Free Edition this week?

[FREE WEBINAR] Running Supply Chain Operations on Databricks: From Dashboards to Agents (BrickTalk)

Hey r/Databricks! We're hosting a free community sponsored BrickTalk this Thursday, May 7th, focusing on modern Supply Chain Management using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with data and AI, giving customers a direct line to the people behind the products. Discover how to build a unified Control Tower that delivers real-time inventory visibility, AI-powered demand forecasting, and autonomous planning. We'll demo an end-to-end operational platform featuring Databricks AI/BI Genie Rooms and multi-agent Supervisor workflows. You'll see: * Dashboards surfacing key insights. * Genie answering natural language queries grounded in live data. * Agentic systems autonomously processing inbound requests to generate fulfillment plans. This is a great chance to see real-world use cases and get practical insights directly from Databricks experts. **When:** Thursday, May 7 * 9:00 am PT * 12:00 pm ET * 5:00 pm London * 9:30 pm IST [Register here](https://usergroups.databricks.com/events/details/databricks-user-groups-bricktalks-presents-supply-chain-management-bricktalk-running-supply-chain-operations-on-databricks-from-dashboards-to-agents/) Drop any questions below! 👇

ai_classify() - Am I going insane?

I need someone to double check me here because I'm hoping I'm confused. So Databricks has it's nice no-code data classification tool in the UI to let users orchestrate AI functionality. Very cool so far. However, one small problem - there's no option in the UI to select a model. It just runs the default offering from the model reg. So I think maybe that's just not implemented yet and go the SQL route to use ai\_classify() directly. Lo and behold you can't choose the model here either, and the built in model choice is still opaque! What? Databricks doesn't seem to do any analysis on my workflow to automatically route to an appropriate model. Is it really just using the same model regardless of prompt complexity? If this is the case, what is the actual usecase for this functionality? I have been cheerleading Databricks at work for a while now and have onboarded a ton of users, but how can I endorse using ai\_classify() in workflows when the spend has the potential to be so wildly mismatched to the task? I'm currently advising people who want low code agentic solutions to use n8n, which is a shame because pretty much all the other resources they might want to include are in Databricks. Of course technical users can use ai\_query(), but the overlap of people who want to code their own agentic workflows and people who want to use Databricks built-ins is pretty small.