Workflows
Recent items mentioning Workflows across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Modern Databricks pipeline architectures are centering on Lakeflow Connect for CDC, Spark declarative streaming, and Unity Catalog governance 3. Meanwhile, dbt released updates spanning autonomous agents, Fusion, and developer experience for Databricks workflows 1, complemented by Databricks guidance clarifying how to delineate LLM versus broader AI tooling across data pipelines 2.
Generated daily from the 3 most recent items mentioning Workflows. Click any [N] to jump to the source.
ADF vs Databricks Workflows — When to Use Which
Databricks Apps vs Power Apps -/ which to use when you need a UI for business rules? (cost-sensitive)
Hi everyone, I’m deciding between Databricks Apps and Microsoft Power Apps for a simple but important need and I’d love your experience/advice. Context: • I have models and data processing in Databricks (already built a demo + deployment using Databricks). • Need a small UI so business users can edit/add rules (stored in a table) that our models use. • I care a lot about cost. I don’t have an easy way right now to start/stop Databricks compute to save money. • I already run regression tests and understand Databricks workflows, but I don’t fully know the real benefits of Databricks Apps vs Power Apps for this use case. Questions: 1. For a rules-editing UI (CRUD for a rules table) what worked better for you: Databricks Apps or Power Apps? 2. Cost-wise, which is cheaper to build/run for low-traffic business users? 3. Any recommended patterns to connect a low-code UI (Power Apps) to Databricks compute securely and cheaply? 4. If you chose Databricks Apps, how do you reduce runtime cost — do you use job clusters, serverless endpoints, or something else? I have this decision tree (WIP).
What's shipped in dbt — May 2026
May 2026 brings a roundup of dbt shipments since January, covering agents, Fusion, security, developer experience, dbt Core, and more. This post details all the product changes relevant to your Databricks workflows.
Databricks is now supporting Microsoft Outlook in Lakeflow Connect (Beta)
Azure Databricks has introduced a managed Microsoft Outlook connector for Lakeflow Connect, currently available in Beta, enabling organizations to ingest Outlook email data directly into Azure Databricks. With this new connector, teams can now integrate Outlook-based communication data into analytics, governance, automation, and AI workflows more efficiently. https://preview.redd.it/8r0xry5yz50h1.png?width=1402&format=png&auto=webp&s=78ae55a64c3df3d47b0edcb4ed6f246f421e7b08 Key capabilities currently supported: * Incremental ingestion * Unity Catalog governance * UI & API-based pipeline authoring * Databricks Workflows orchestration * Declarative Automation Bundles * Column selection/deselection Supported authentication: * OAuth M2M (Machine-to-Machine) Current Beta limitations: * SCD Type 2 support * Automated schema evolution * API-based row filtering * Multiple tables per pipeline (currently limited to 1) Since the connector is still in Beta, workspace admins must enable the feature from the Previews page before use. Nice to see Databricks continuing to expand Lakeflow Connect integrations across enterprise ecosystems. [Source Link](https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/outlook-overview)
LLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools
LLMs are a subset of AI, and this guide clarifies their practical differences, use cases, and tools. Understand how LLMs fit into the broader AI landscape and what that means for your Databricks workflows.
Transitioning from ADF to Databricks Workflows: Best Practices in a Multi-Workspace (dev-prod)
TutorialsDatabricks End-To-End Project | Zero-To-Expert | Streaming, AI, Lakeflow, Unity Catalog, AI/BI
This video demonstrates building an end-to-end restaurant analytics platform on Databricks, covering streaming and batch data ingestion, AI-powered sentiment analysis, and dashboard creation. It teaches how to use Unity Catalog, Lake Flow Connect for CDC, Spark declarative pipelines for real-time data from Event Hub, and how to construct a medallion architecture with fact and dimension tables.
TutorialsOrchestration With Lakeflow Jobs
Lakeflow Jobs is a native orchestrator for Databricks that automates data workflows through scheduling, data-triggered execution, and built-in observability without requiring external tools. The presentation demonstrates building an end-to-end ETL pipeline with data ingestion from Salesforce, transformation via Delta Live Tables, and automated dashboard updates, all configured through a visual UI.
NewsFrom Days to Seconds — Reducing Query Times on Large Geospatial Datasets by 99%
The Global Water Security Center reduced geospatial query times from days to seconds by implementing Databricks medallion architecture with H3 spatial indexing and autoloader data ingestion on 30-billion-row datasets. This achieved a 99% query time reduction, saving approximately $1 million annually in labor costs and enabling analysts to self-serve analysis workflows through automated notebook orchestration.
NewsScaling Data Engineering Pipelines: Preparing Credit Card Transactions Data for Machine Learning
Delta Lake's partitioning, z-ordering, and file compaction reduced Mastercard's 10+ petabyte transaction table file count by 70% and improved query performance by 80%. Databricks Workflows' foreach operator enabled parallel processing across multiple clusters, completing nine months of graph network computation in two days.
NewsCreating a Custom PySpark Stream Reader with PySpark 4.0
The video demonstrates how to create a custom PySpark 4.0 streaming data source by implementing Python classes that inherit from the new data source interfaces. It walks through building a custom stream reader to ingest data from an unsupported messaging system like Apache Active MQ directly into Delta tables.
Added migrate-dlt-pipelines command for Delta Live Tables migration from HMS to UC and expanded HMS Federation to support MSSQL and PostgreSQL. Fixed schema skip/unskip functionality and enhanced local code migration with automatic fixing capabilities via the improved migrate-local-code command.
EventsEmbracing the Future of Data Engineering: The Serverless, Real-Time Lakehouse in Action
NewsUS Army Corp of Engineers Enhanced Commerce & National Sec Through Data-Driven Geospatial Insight
Get Tuesday's version of this
Tracking Workflows? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.