AI Functions
Recent items mentioning AI Functions across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks is rolling AI Functions into core SDK tooling, adding workspace service support in both the Go 2 and Java 3 SDKs within a single week (September 3–9). Both releases pair AI Functions support with Sandboxes and Feature Engineering backfill methods, but also ship breaking changes stripping fields from catalog and model-serving configurations 23.
Generated daily from the 4 most recent items mentioning AI Functions. Click any [N] to jump to the source.
Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?
Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ai_query ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]
Release introduces three new workspace services (AI Functions, Domains, Sandbox), extends Feature Engineering with new backfill and operation management methods, and adds AWS Secrets Manager and Azure Key Vault connection support. Multiple breaking changes remove fields from model serving and catalog configurations that may require code updates.
This release adds SDK support for AI Functions and Sandboxes, as well as new backfill management methods for Feature Engineering. Breaking changes remove the owner field across multiple catalog model services, along with trafficSplitting from model routing and disabled from inference table configurations.
NewsDatabricks LakeFlow Designer: Visual Pipelines, Real Code
LakeFlow Designer is a visual no-code tool for building governed data pipelines that generates transparent SQL or Python code. Recent releases add AI-powered operator search, on-demand custom operators, direct table transformations, email notifications, and built-in visualizations.
Databricks: 5 Minute Features - AI Functions
To celebrate that AI functions are now a fully governed part of Unity Catalog alongside the rest of your functions I thought it would be fun to do a run through of how easy it is to use. submitted by /u/Remarkable_Rock5474 [link] [comments]
Tagging individual ai_query() calls in SQL?
Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps. We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step. From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL. Has anyone found a workaround, or is moving the calls out of SQL currently the only option? Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc submitted by /u/hulioshort [link] [comments]
Ingest semi-structured data faster and more efficiently with Variant - Now Generally Available
Databricks Variant is now Generally Available, enabling teams to achieve up to 30x faster reads on semi-structured data while handling unpredictable schema changes without pipeline updates. The feature is broadly integrated across the platform, supporting data workloads like Auto Loader and Spark Declarative Pipelines alongside AI tools like Agent Bricks and AI Functions.
Transforming solar and wind maintenance reports with Genie and AI agents
Plenitude now converts unstructured solar and wind maintenance PDFs into a unified, queryable data model using Databricks Genie and AI agents. This enables natural-language querying and visualizations across plants, accelerating multi-plant analysis and laying the groundwork for predictive maintenance.
TutorialsThe New Databricks Lakeflow Designer Is a Game Changer!
Databricks Lakeflow Designer is a visual data preparation tool that allows users to create, add, and transform data using a no-code drag-and-drop UI or AI-powered Genie Code. The video demonstrates how to import data from various sources, profile data, perform complex transformations like data type conversions and sentiment analysis, and then deploy the resulting production-ready PySpark code for scheduling or integration into existing pipelines.
NewsMay 2026 Databricks Updates: No Code ETL, New GPUs and Death of the Dashboard
Databricks announced several updates including AI Prep Search for document chunking and vector database preparation, SQL vector functions for embedding mathematics, and the general availability of multi-table transactions. They also introduced Lakeflow Designer for visual, no-code data pipeline creation and updated their serverless GPU offerings to include H100s.
ReleasesIntroducing Databricks Document Intelligence
Databricks Document Intelligence is a new solution for extracting, processing, and analyzing unstructured data from documents using large language models. It offers a unified platform for document processing, including data extraction, summarization, and question answering, with a focus on accuracy and scalability.
Additional thoughts after spending about 10 hours trying the new Lakeflow Designer
After spending some more time on Lakeflow Designer last night/into the early hours of today, it is now my favorite feature from Databricks this year, and I really hope Databricks builds even more features around it. Some thoughts: \-Just about everything in this pipeline shown was "vibed", but every step has a visual representing the flow of data in a way that technical and tech-savvy business folks can easily audit and/or modify as needed. \-The ability to use the AI functions like ai\_classify, ai\_summarize, and others is the cleanest for discovery, testing, and getting things into production. AI functions for everyone. Don't even need to know SQL! \-Being able to easily bring in data from Excel into a governed environment to be joined with governed tables is very practical, representing how things happen in the real world. And of course, being able to export back to Excel is also nice (or connect via the Excel connector!). \-Is it 100% ready to beat all the tools out there in this category? No. Will it eventually get there? I believe so and more. Today, you can already begin to do a lot with it. \-If you do try it out and wish it had some other capabilities, I highly encourage you to share your feedback with Databricks as I know they are actively listening to make this product beneficial for a broad range of customers. The fact that there are already more than a handful of videos out there showcasing Lakeflow Designer tells me many others are very excited about this as well. In the next few days, I'll be sharing a video or two of my own around this.
Unity Catalog AI 0.4.0
DatabricksFunctionClient now accepts an optional warehouse_id parameter to execute functions via the Statement Execution API when serverless compute is unavailable, while serverless remains the default execution path. The release requires Python 3.10 or later and fixes issues with Gemini toolkit, LangGraph integration, OSS client function creation, and dependency declarations.
UnityCatalog 0.4.0
Unity Catalog 0.4.0 introduces AWS storage credentials and external locations enabling secure S3 access with automatic temporary credential vending, and makes CREATE TABLE AS SELECT atomic for Delta tables to prevent data corruption. Credential renewal defaults to enabled for long-running Spark jobs, and new DSPy integration allows UC AI functions to serve as tools in LLM agent workflows.
Get Tuesday's version of this
Tracking AI Functions? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.