Databricks Pipelines: Write to One Unity Catalog Table from Multiple Sources with FLOWs
Description
Learn how to write to a single managed Unity Catalog table from multiple data sources using flows in Databricks pipelines. WHAT YOU'LL LEARN • Single Table, Multiple Sources: Write from multiple independent streams into one target table within the same pipeline • Independent Offsets: Track progress per flow with stable flow-name checkpoints • Zero-Reprocessing Migration: Migrate existing Structured Streaming jobs seamlessly using import_checkpoint in Python TIMESTAMPS 0:00 - One table, multiple sources, one pipeline 0:11 - Create the managed target (SQL & Python) 0:17 - Add one flow per source 0:28 - Stable flow names & checkpoints 0:35 - Migrate Structured Streaming (import_checkpoint) 0:50 - Checkpoint copying & resume 0:59 - Complete SQL syntax example
Description from YouTube. Full content on the video page.
More from Databricks
NewsHow Repsol Democratizes AI With Databricks Genie | Data + AI Executive Series: Fast 5
NewsBuilding Reliable Data Pipelines with Lakeflow on Databricks | Lakeflow Demo
Databricks Lakeflow provides a data intelligence platform for building and managing reliable data pipelines. The demonstration covers creating delta tables, ingesting data from Salesforce using Lakeflow Connect, and using Lakeflow Designer for data transformation and orchestration.
TutorialsBuilding an Agentic Analytics Foundation on the Lakehouse
The video demonstrates how to build an agentic analytics foundation on the Databricks lakehouse platform to ensure AI agents are accurate, secure, and scalable. It explains how to organize data assets into business domains, define metric views, and use Unity Catalog and Unity Gateway for governance and guardrails.
NewsLG Uplus moves from static analytics to real-time AI | Data + AI Executive Series: Fast 5
LG Uplus uses Databricks to operationalize AI and address siloed data challenges across the telecom enterprise. The company integrates AI into core workflows to deliver automated, data-driven customer experiences.
NewsTrain and fine-tune models faster with AI Runtime
Databricks AI Runtime is a managed GPU experience for distributed training and fine-tuning of AI models at scale. The video demonstrates using AI Runtime to train and fine-tune a model and then analyze the run to improve model quality and efficiency.