No More Fragile Pipelines: Kafka and Iceberg the Declarative Way
Summary
Kafka topics and Iceberg tables have incompatible data models that cause problems with schema evolution, consistency, and file fragmentation when building data pipelines. Confluent's Table Flow solution declaratively handles schema evolution, manages consistency through two-phase commits with segments, and performs automatic compaction to optimize query performance.
Summary generated by brickster.ai from the video transcript.
More from Databricks
NewsBuilding Reliable Data Pipelines with Lakeflow on Databricks | Lakeflow Demo
Databricks Lakeflow provides a data intelligence platform for building and managing reliable data pipelines. The demonstration covers creating delta tables, ingesting data from Salesforce using Lakeflow Connect, and using Lakeflow Designer for data transformation and orchestration.
TutorialsBuilding an Agentic Analytics Foundation on the Lakehouse
The video demonstrates how to build an agentic analytics foundation on the Databricks lakehouse platform to ensure AI agents are accurate, secure, and scalable. It explains how to organize data assets into business domains, define metric views, and use Unity Catalog and Unity Gateway for governance and guardrails.
NewsLG Uplus moves from static analytics to real-time AI | Data + AI Executive Series: Fast 5
LG Uplus uses Databricks to operationalize AI and address siloed data challenges across the telecom enterprise. The company integrates AI into core workflows to deliver automated, data-driven customer experiences.
NewsTrain and fine-tune models faster with AI Runtime
Databricks AI Runtime is a managed GPU experience for distributed training and fine-tuning of AI models at scale. The video demonstrates using AI Runtime to train and fine-tune a model and then analyze the run to improve model quality and efficiency.
TutorialsTutorial: Genie for Data Engineers, Data Scientists, and Everybody Else
This tutorial demonstrates how to use Databricks Genie to build a live web app and data pipeline from a 700-million-row IoT dataset. The process covers getting the data from the Databricks Marketplace, running exploratory data analysis, building pipelines and jobs with Genie Code, querying data on mobile, and receiving shared data locally.
