AI Functions
Recent items mentioning AI Functions across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks rolled out coordinated breaking changes to AI Functions batch processing across its Go, Java, and Python SDKs 234. The batch output reference field is now mandatory, and the expired status enum value has been removed across all three language clients 234.
Generated daily from the 6 most recent items mentioning AI Functions. Click any [N] to jump to the source.
The SDK adds the workspace-level AssumeGroup IAM service and an EffectiveResources field for apps. Breaking changes make the OutputRef field required and remove the Expired status in AI Functions batches, alongside dropping the Continuous schedule field in Lakewatch.
This release adds support for assuming groups in the workspace client and introduces an effective resources field for Databricks Apps. Breaking changes require the AI Functions batch output reference field while removing the continuous rule schedule field and the expired batch status enum value.
The release introduces the workspace-level assume_group service, adds an effective_resources field for Apps, and ensures SDK User-Agent headers are sent on all Files API requests. Breaking changes make output_ref required for AI Functions batches, remove the continuous schedule in LakeWatch, and drop the expired status enum value.
How I check every prompt before the LLM on Databricks: GovTech's LionGuard on a small CPU endpoint
I wanted one shared safety check that every agent, job or SQL query on Databricks calls before the LLM. How it works: - LionGuard 2 Lite from GovTech Singapore: a small classifier on Google's EmbeddingGemma. It scores insults, hate, violence, self-harm and misconduct, and it handles English, Singlish, Chinese, Malay and Tamil. - Packaged as one MLflow model, registered in Unity Catalog, served on a small CPU endpoint. - From SQL, ai_query sends each question to the guard. Only questions with an unsafe score under 0.5 go to the LLM. Results on 7 synthetic questions: a Singlish insult, a Malay threat, a harmful Chinese message and an English "how do I hack the website" question were blocked. Safe questions in English, Singlish and Tamil were answered. Lessons: - The full set-up took 25 minutes. 19 of them were the endpoint start. - Do not use scale to zero for a production guard (cold start). - Turn on inference tables from day one, so you can show what was blocked. - It does not detect prompt injection. That needs a separate check. How do you check prompts for your agents today? Built-in AI Gateway guardrails, a custom endpoint, or both? submitted by /u/sundarleads [link] [comments]
AI Functions for Synthetic Test Data: Generate Realistic Datasets in Pure SQL
Unlock powerful insights and forecasts with Predictive AI Functions
Databricks has introduced five Predictive AI Functions, featuring an upgraded ai_forecast alongside the new ai_top_drivers, ai_detect_anomalies, ai_predict_value, and ai_predict_class. These tools allow SQL analysts and business users to generate predictions, detect anomalies, and uncover metric drivers directly within Genie and existing workflows without building or maintaining models.
Turning P&C Claim Documents into Structured Data with Databricks AI Functions
Databricks ai_decide: The AI Judge That Never Writes a Word
Use Databricks ai_decide() to grade every ai_query output in SQL and decide what gets sent, without a second LLM submitted by /u/Lenkz [link] [comments]
Routing between GLM 5.3 Flash and GLM 5.3 with ai_decide: a cheap router in front of ai_query
The AI functions service now supports an ai_decide method, and multiple Jobs and Tasks models now accept environment variable configurations. Additionally, command_path in DeploymentSpec and api_secret_ref in SchemaRegistryConfig are no longer required fields.
Running open-Jev in SQL on Databricks
Run models like SemIf-OpenJev directly on governed Databricks data using serverless GPUs and managed Model Serving. Query custom endpoints from SQL or Lakeflow jobs using ai_query to return structured decisions and probability scores at scale, with an importable notebook to deploy and adapt the workflow.
AI is apparently getting a new name: SI (Super Intelligence)
Donald Trump said at the UN that super-intelligence will replace artificial intelligence in US government documents. If you work with Databricks, this probably sounds strangely familiar. So now I’m wondering… was Databricks just preparing for this? Because there are still plenty of names left: AI Search → SI Search? AI Functions → SI Functions? AI Runtime → SI Runtime? AI Gateway → SI Gateway? P.S. I keep track of Databricks renames and product changes on REbricked.org submitted by /u/Significant-Guest-14 [link] [comments]
Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?
Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ai_query ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]
Release introduces three new workspace services (AI Functions, Domains, Sandbox), extends Feature Engineering with new backfill and operation management methods, and adds AWS Secrets Manager and Azure Key Vault connection support. Multiple breaking changes remove fields from model serving and catalog configurations that may require code updates.
This release adds SDK support for AI Functions and Sandboxes, as well as new backfill management methods for Feature Engineering. Breaking changes remove the owner field across multiple catalog model services, along with trafficSplitting from model routing and disabled from inference table configurations.
Enhancing Agent Retrieval with Structured Chart Extraction
Extracting charts as structured JSON with ai_parse_document improves document retrieval and agent answer accuracy on chart-based questions while outperforming large multimodal embedding models. By indexing these extractions with lightweight 300M-parameter text embeddings via Databricks AI functions, practitioners can enable tools like Databricks Genie to reliably resolve fine-grained numerical queries from visual documents.
Databricks: 5 Minute Features - AI Functions
To celebrate that AI functions are now a fully governed part of Unity Catalog alongside the rest of your functions I thought it would be fun to do a run through of how easy it is to use. submitted by /u/Remarkable_Rock5474 [link] [comments]
Tagging individual ai_query() calls in SQL?
Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps. We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step. From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL. Has anyone found a workaround, or is moving the calls out of SQL currently the only option? Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc submitted by /u/hulioshort [link] [comments]
Using AI_Functions in Your Data Warehouse: Top Use Cases
Databricks AI Functions bring model inference directly into your data warehouse using pure SQL, replacing external APIs and brittle Python scripts for tasks like document parsing, translation, and text classification. By executing functions like ai_parse_document and ai_classify natively inside existing query plans, teams can process unstructured data without shipping it outside Unity Catalog governance.
Ingest semi-structured data faster and more efficiently with Variant - Now Generally Available
Databricks Variant is now Generally Available, enabling teams to achieve up to 30x faster reads on semi-structured data while handling unpredictable schema changes without pipeline updates. The feature is broadly integrated across the platform, supporting data workloads like Auto Loader and Spark Declarative Pipelines alongside AI tools like Agent Bricks and AI Functions.
Transforming solar and wind maintenance reports with Genie and AI agents
Plenitude now converts unstructured solar and wind maintenance PDFs into a unified, queryable data model using Databricks Genie and AI agents. This enables natural-language querying and visualizations across plants, accelerating multi-plant analysis and laying the groundwork for predictive maintenance.
Unity Catalog AI 0.4.0
DatabricksFunctionClient now accepts an optional warehouse_id parameter to execute functions via the Statement Execution API when serverless compute is unavailable, while serverless remains the default execution path. The release requires Python 3.10 or later and fixes issues with Gemini toolkit, LangGraph integration, OSS client function creation, and dependency declarations.
UnityCatalog 0.4.0
Unity Catalog 0.4.0 introduces AWS storage credentials and external locations enabling secure S3 access with automatic temporary credential vending, and makes CREATE TABLE AS SELECT atomic for Delta tables to prevent data corruption. Credential renewal defaults to enabled for long-running Spark jobs, and new DSPy integration allows UC AI functions to serve as tools in LLM agent workflows.
Get Tuesday's version of this
Tracking AI Functions? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.