Kafka

Learn about Zerobus in 15 min!

Databricks Lakeflow Connect Zerobus Ingest is a high-performance, multi-cloud ingestion service that allows users to stream event data directly into their lakehouse without the cost and complexity of a traditional message bus. The video explains the architecture of Zerobus Ingest, announces upcoming API integrations for Kafka and MQTT, and demonstrates how to configure and run a Python client to write data directly into a Delta table.

How an Open, Scalable and Secure Data Platform is Powering Quick Commerce Swiggy's AI

Swiggy uses a Databricks-based lakehouse platform with Delta Lake to manage billions of daily events across 700+ cities, supporting real-time delivery predictions and resource optimization at 10-minute delivery speeds. The platform enables AI democratization through data agents and chatbots that provide automated customer support and business intelligence without requiring SQL expertise.

No More Fragile Pipelines: Kafka and Iceberg the Declarative Way

Kafka topics and Iceberg tables have incompatible data models that cause problems with schema evolution, consistency, and file fragmentation when building data pipelines. Confluent's Table Flow solution declaratively handles schema evolution, manages consistency through two-phase commits with segments, and performs automatic compaction to optimize query performance.

Streaming Meets Governance: Building AI-Ready Tables With Confluent Tableflow and Unity Catalog

Confluent Tableflow automatically exposes Kafka streaming topics as Delta tables and syncs them to Databricks Unity Catalog, eliminating manual data pipeline work. A live demo shows how this enables immediate real-time analytics on stock market data streams without traditional ETL preprocessing steps.

Sponsored by: Confluent | Turn SAP Data into AI-Powered Insights with Databricks

Confluent streams SAP operational data, enriches it with real-time context, and feeds it to Databricks for machine learning and analytics. The platform enables bidirectional data flow between operational and analytical systems, allowing AI agents to access both historical data and fresh signals while returning insights back to production applications.

Creating a Custom PySpark Stream Reader with PySpark 4.0

The video demonstrates how to create a custom PySpark 4.0 streaming data source by implementing Python classes that inherit from the new data source interfaces. It walks through building a custom stream reader to ingest data from an unsupported messaging system like Apache Active MQ directly into Delta tables.

Practical Pipelines: A Houseplant Alerting System with ksqlDB

This video demonstrates how to build a real-time houseplant monitoring system using Apache Kafka and a Raspberry Pi. It teaches how to use ksqlDB for stateful stream processing, configure Kafka topics, and integrate hardware sensors with external messaging alerts.

Unlocking Near Real Time Data Replication with CDC, Apache Spark™ Streaming, and Delta Lake

DoorDash implemented an open-source near real-time data ingestion framework called Pepto using Apache Kafka, Apache Spark Streaming, and Delta Lake. The system captures change data feeds from various transactional databases and performs micro-batched upserts to achieve a P99 data latency of 7 to 30 minutes.

Connecting the Dots with DataHub: Lakehouse and Beyond

Near Real-Time Analytics with Event Streaming, Live Tables, and Delta Sharing

Optimizing Speed and Scale of User-Facing Analytics Using Apache Kafka and Pinot

Backfill Streaming Data Pipelines in Kappa Architecture

Serverless Kafka and Apache Spark in a Multi-Cloud Data Lakehouse Architecture

Streaming Data into Delta Lake with Rust and Kafka

Automating Business Decisions Using Event Streams

How Adobe uses Structured Streaming at Scale

Evolution is Continuous, and so are Big Data and Streaming Pipelines

How Adobe Does 2 Million Records Per Second Using Apache Spark!

Automating Federal Aviation Administration's System Wide Info Management Data Ingestion and Analysis