LakeFlow
Recent items mentioning LakeFlow across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Declarative Pipelines dominate the current cycle, spanning edge cases 1, serverless triggered-vs-continuous execution 3, and a testing blueprint for moving them to production 10. Lakeflow Connect reliability issues surfaced repeatedly: NetSuite transactionline refreshes stuck at 3.5–4.8 hrs 5 and SQL Server gateway Entra ID auth failures (error 18456, state 132) 7. Databricks also pushed Lakeflow toward non-engineers with the no-code Lakeflow Designer for visual pipeline building 9.
Generated daily from the 10 most recent items mentioning LakeFlow. Click any [N] to jump to the source.
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow
Triggered vs. Continuous Mode: A Deep Dive into Serverless Lakeflow Spark Declarative Pipelines
Feedback on Deploy Workloads with Lakeflow Jobs SPL
NetSuite connector (Lakeflow Connect) — transactionline refresh takes 3.5–4.8 hrs regardless
Lakeflow Connect - Community Custom Connector - How to troubleshoot source ingestion logic "live"
Lakeflow Connect SQL Server gateway – intermittent Entra ID token auth failures (18456, state 132)
NewsNo-Code Databricks: Genie Agents, Genie Code, Lakeflow Designer Explained
Databricks has transformed from a code-heavy platform into a no-code solution for business users, eliminating barriers to self-service analytics. Lakeflow Designer enables visual drag-and-drop data pipeline building while AI Genie allows querying data in plain English to instantly generate interactive dashboards.
From Experiment to Prod: LakeFlow Spark Declarative Pipelines Testing Blueprint
Announcing General Availability of the Lakeflow Spark Declarative Pipelines Kafka Sink
Simplify AI agent orchestration with Lakebase Postgres
Learn how to turn two Lakebase Postgres tables into a durable, crash-resilient queue for long-running AI agent tasks without operating an external broker, cache, or scheduler. This fully Databr
Issue: Lakeflow Connect Microsoft Teams Community Connector - No module named 'databricks.labs'
INSERTS AND DELETES in a massive way for Lakeflow Spark Declarative Pipelines
sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?
Handling Sensor Dropout in IoT Pipelines: A Quarantine Pattern with Lakeflow Declarative Pipelines
Lakeflow Community Connector Issue - ModuleNotFoundError: No module named 'databricks.labs'
What is the compute for Lakeflow Connect SharePoint Connector
Lakeflow Connect & Community Connectors - 403 Errors + What is the compute?
Query Federation vs Lakeflow Connect in Databricks: When to Query and When to Ingest
Building a soccer coaching app on Databricks
Coach's Corner is a Databricks App that processes 25 fps match tracking data into a sub-second 2D/3D tactical bench with replays, event analytics, a scout chat, and an opponent-dossier agent. The end-to-end solution is powered entirely on the Databricks platform, utilizing Lakeflow pipelines for data refinement, DBSQL and Lakebase for rapid querying, and Unity Catalog-governed AI tools like Genie, Vector Search, and MLflow tracing.
AI-Powered Data Engineering with Lakeflow: Techniques for Modern Data Professionals | Virtual Event
Lakeflow Connect SharePoint - 403 Error
NewsDatabricks News: RT Lakehouse (Reyden), Lakebase, TTL
This video highlights recent Databricks updates, including the beta release of the high-performance "Raiden" real-time lakehouse engine and new lakeflow connectors. It also demonstrates administrative changes to user groups, new time data types, predictive optimization TTL deletes, user home volumes, and advanced search capabilities in Lakebase.
NewsLearn about Zerobus in 15 min!
Databricks Lakeflow Connect Zerobus Ingest is a high-performance, multi-cloud ingestion service that allows users to stream event data directly into their lakehouse without the cost and complexity of a traditional message bus. The video explains the architecture of Zerobus Ingest, announces upcoming API integrations for Kafka and MQTT, and demonstrates how to configure and run a Python client to write data directly into a Delta table.
TutorialsMastering Joins In Apache Spark: Complete Deep Dive
The video provides a deep dive into four Apache Spark physical join strategies: Sort Merge Join, Broadcast Hash Join, Shuffle Hash Join, and Broadcast Nested Loop Join. For each join, it explains the conditions for Spark's selection, visualizes its step-by-step internal mechanics, and demonstrates its appearance in Spark's physical plan and UI.
How Databricks is turning video into searchable, actionable intelligence
Databricks now enables transforming raw video into searchable, AI-ready intelligence for public sector agencies. This is achieved through VLMs, serverless GPUs, and Lakeflow pipelines to automatically detect, truncate, and summarize key video moments, supporting real-time analysis for public safety and infrastructure.
NewsWhat’s coming next to Free Edition
Databricks announces the availability of Genie, GPUs, Agent Hooks, Lakehouse, and Lake Flow Designer on its Free Edition. This update provides virtually all of Databricks' production platform features for free, enabling users to learn and build data and AI projects.
EventsUnlocking agentic data engineering with Lakeflow + Genie
The video introduces Lakeflow as a unified, open data engineering stack that simplifies data transformation, ingestion, and orchestration through declarative pipelines, no-code tools, and managed services. It also announces Genie Zero Ops, an AI agent that automates data operations by autonomously detecting, diagnosing, and verifying fixes for data incidents and PII exposures within the data plane.
EventsRecap of product announcements from Data + AI Summit 2026 | Day 1
Databricks announced several new products and features at the Data + AI Summit 2026, Day 1, including the Genetic Data Foundation, Lakehouse RT, Lake Base with disaster recovery, Lake Flow, Genie Ontology, Unity AI Gateway, Omnigent, and various Genie agents (Genie 1, Genie Code, Genie Agents). They also introduced new applications like Lake Watch for SIM and Customer Lake for CP.
What’s coming next to Free Edition
Databricks Free Edition now includes every core practitioner feature, expanding with Genie Code, GPUs, Lakebase, Lakeflow Designer, and Agent Bricks. This gives users a complete, free toolkit for building end-to-end data and AI projects.
What is data pipeline architecture?
Data pipeline architecture separates ingestion, transformation, storage, and serving into distinct layers, with ELT largely replacing ETL as the dominant approach. Databricks unifies batch and streaming pipelines on a single platform (Lakeflow + Delta Lake + Unity Catalog), eliminating duplicate infrastructure and governance gaps.
Lakeflow: A new era of agentic data engineering
Lakeflow unifies ingestion, transformation, and orchestration under Unity Catalog, providing a single source of trusted, real-time context for agentic AI. It offers high-performance ingestion from 100+ sources, real-time streaming, visual pipeline building with Lakeflow Designer, and AI-powered authoring and operations with Genie Code and Genie ZeroOps.
EventsDatabricks News: CLI v 1.0.0, AI-tools, databricks Docker, DABs UI sync, mutators
The video demonstrates new Databricks features, including the GA release of CLI 1.0.0, UI sync for DABs, Python mutators for bundle extension, and new Docker image options for custom runtimes. It also covers serverless pipeline orchestration, enhanced autoscaling for Lakebase and apps, serverless interactive execution timeout, and auto-scoping for access tokens.
TutorialsPro Tip: Add Multiple Tables Fast! 🚀
Users can quickly add multiple tables to a canvas by dragging them directly from the Catalog Explorer left panel. This method streamlines the process of adding several tables from the same schema or catalog, avoiding the need to create individual source nodes.
Databricks Asset Bundles now features a select flag for plan and deploy commands to target specific resources alongside their transitive dependencies. The experimental open command now supports all bundle resource types with workspace URLs
TutorialsThe New Databricks Lakeflow Designer Is a Game Changer!
Databricks Lakeflow Designer is a visual data preparation tool that allows users to create, add, and transform data using a no-code drag-and-drop UI or AI-powered Genie Code. The video demonstrates how to import data from various sources, profile data, perform complex transformations like data type conversions and sentiment analysis, and then deploy the resulting production-ready PySpark code for scheduling or integration into existing pipelines.
TutorialsThe Future of Finance Operations Starts Here
The video demonstrates how Databricks' financial lakehouse solution addresses common finance data challenges like fragmentation and slow analysis. It showcases features like Unity Catalog for data governance, Lake Flow for pipeline management, and Genie Spaces for natural language querying of financial data.
EventsDatabricks News: Lakeflow Designer, UV package manager, DABs templates, Genie scheduled tasks
Databricks introduces Lakeflow Designer for visual data preparation, though its generated code is messy; a workaround uses Genie to convert the visual workflow into clean PySpark/SQL notebooks. The UV package manager significantly speeds up package installations on Databricks serverless runtimes, and DABs templates allow for standardized, customizable Databricks Asset Bundles.
NewsMay 2026 Databricks Updates: No Code ETL, New GPUs and Death of the Dashboard
Databricks announced several updates including AI Prep Search for document chunking and vector database preparation, SQL vector functions for embedding mathematics, and the general availability of multi-table transactions. They also introduced Lakeflow Designer for visual, no-code data pipeline creation and updated their serverless GPU offerings to include H100s.
The Rise of Sports Intelligence: How the Lakehouse Turns Tracking Data into Competitive Advantage
Pro teams now leverage the Lakehouse to transform exploding tracking and biomechanical data into sports intelligence, driving real-time decisions on the court, in training, and in the front office. The Databricks Data Intelligence Platform acts as the governed "sports brain," unifying diverse data with Lakeflow, Unity Catalog, ML, and AI Search to power proactive injury management, coaching insights, and next-gen fan experiences.
ReleasesIntroducing Databricks Document Intelligence
Databricks Document Intelligence is a new solution for extracting, processing, and analyzing unstructured data from documents using large language models. It offers a unified platform for document processing, including data extraction, summarization, and question answering, with a focus on accuracy and scalability.
NewsDatabricks News: watermark-based incremental ingestion, MCP in AI gateway, Genie, Vector Search
Databricks now offers watermark-based incremental ingestion from SQL databases without change data feed, allowing for efficient data updates and soft deletion handling. The AI Gateway supports custom MCP servers, enabling integration with external APIs like GitHub for enhanced AI application development.
Agentic Data Engineering with Genie Code and Lakeflow
Genie Code, an autonomous AI partner for data engineers, is now integrated directly into Lakeflow. Data engineers can leverage Genie Code within Lakeflow's Pipeline Editor and Jobs for the full data engineering lifecycle, from development and orchestration to monitoring and debugging.
Announcing the Public Preview of Lakeflow Designer
Lakeflow Designer is now in Public Preview, offering a visual, no-code, AI-native interface for data preparation and analysis directly within Databricks. It leverages Unity Catalog for governance and generates production-ready code, providing step-by-step data previews for easier review of AI-generated transformations.
Tutorials54 Zerobus Ingest Lakeflow Standard Connector | Ingest Streaming data directly into Delta Table
The video demonstrates how to use Databricks Zero Bus Ingest, a push-based API, to directly stream various data types like IoT, event, and telemetry data into Unity Catalog Delta tables. It highlights Zero Bus Ingest's ability to simplify streaming ingestion by eliminating the need for intermediate message buses and managing their infrastructure.
NewsDatabricks News: Free Tier, Multi-statement transactions, Declarative Automation Bundles, Genie Code
Databricks now offers a free tier for Lakeflow Connect, providing 100 DBUs per day per workspace, and has introduced multi-statement transactions in Unity Catalog that ensure atomicity with rollback capabilities. The platform also announced a Databricks One mobile app, a new AI runtime with pre-installed tools for GPU use cases, and enhanced Genie Code that understands project structure for automated development tasks. Additionally, Databricks Asset Bundles are now called Declarative Automation Bundles and use a faster direct engine, and a new 5X-Large SQL warehouse is available for processing terabytes of data.
TutorialsDatabricks AI Dev Kit Demo - Install, DataGen, SDP, Dashboard
The video demonstrates installing the Databricks AI Dev Kit on a Mac, then uses it to generate synthetic data, create serverless Spark declarative pipelines for a medallion architecture, and build a Databricks dashboard based on the generated data. It highlights how the AI Dev Kit leverages skills and an MCP server to automate these development tasks.
Tutorials53 Lakeflow Connect SQL Server Managed Connector | Ingest Data using Databricks native connectors
The video demonstrates how to ingest data from SQL Server into Databricks using Lakeflow Connect's managed connector, covering the setup of a SQL Server database, user permissions, and enabling change tracking/change data capture (CT/CDC). It then walks through configuring the Databricks connection, creating gateway and ingestion pipelines, and showcasing how SCD Type 2 changes are automatically managed.
TutorialsDatabricks End-To-End Project | Zero-To-Expert | Streaming, AI, Lakeflow, Unity Catalog, AI/BI
This video demonstrates building an end-to-end restaurant analytics platform on Databricks, covering streaming and batch data ingestion, AI-powered sentiment analysis, and dashboard creation. It teaches how to use Unity Catalog, Lake Flow Connect for CDC, Spark declarative pipelines for real-time data from Event Hub, and how to construct a medallion architecture with fact and dimension tables.
NewsDatabricks Breaking News: 2026 Week 3: 12 January 2026 to 18 January 2026
Databricks Runtime 18 is now Generally Available, offering Spark 4.1 and improved identifier/parameter maker availability. New features include Lakeflow Connect for row filtering during ingestion, Codex models (GBT Codex Max and Mini) for code development, and Databricks One improvements like favorites and data preview in Gen Rooms.
NewsVibe-Engineering LakeFlow Pipelines, the Advancing Analytics Way
Advancing Analytics introduces Lake Forge, an engineering framework that uses LLMs and an agentic workflow to generate standardized LakeFlow pipeline templates from data specifications. This system aims to enable scalable, repeatable, and supportable data pipeline creation by balancing AI-driven "vibe coding" with human-engineered guardrails and validation loops.
NewsDatabricks Breaking News: Week 51: 15 December 2025 to 21 December 2025 #databricks news
Databricks introduces new Lakeflow Connect features, including custom logic for declarative pipelines and new connectors for incremental data import from sources like Confluence, PostgreSQL, and MySQL. The platform also announces the deprecation of legacy features like Hive Metastore and DBFS for new accounts, alongside updates to Lakehouse ACLs, job scheduling from notebooks, flexible node types for cluster deployment, and expanded resource assignment in Databricks apps.
NewsDatabricks Breaking News: Week 50: 8 December 2025 to 14 December 2025 #databricks news
Databricks now supports native reading and writing of Excel files in PySpark, SQL, and Autoloader, including features like sheet listing and range targeting. Additionally, Databricks Runtime 18 is available in beta, introducing improvements for streaming queries and new system columns for job tables, alongside a new Legase experience with project and branching capabilities for transactional databases.
Tutorials52 Lakeflow Spark Declarative Pipelines | New Pipeline Code Editor | AUTO CDC |External Target Sinks
Databricks' LakeFlow Spark Declarative Pipelines (SDP), formerly Delta Live Tables (DLT), offers a unified solution for data ingestion, transformation, and orchestration, now open-sourced with Apache Spark 4.1. The video demonstrates using the new pipeline code editor to build SDPs in Python and SQL, showcasing features like auto CDC (formerly apply changes) and external target sinks.
Events[Demo] Lakeflow Designer: No-Code ETL, Powered by the Data Intelligence Platform
Lakeflow Designer allows users to create ETL pipelines using a no-code approach. It features a "transform by example" assistant that can generate data transformations from a screenshot of desired output.
