Structured Streaming
Recent items mentioning Structured Streaming across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Recent developments highlight the growing maturity of Spark's Structured Streaming Real-Time Mode (RTM), demonstrating significant latency reductions. RTM achieves P50 and P95 latencies of 26ms and 50ms respectively in simplified setups 2, and enables sub-second end-to-end latency for complex applications like real-time air traffic control 3. The community is actively exploring RTM for millisecond streaming pipelines with Kafka 1.
Generated daily from the 3 most recent items mentioning Structured Streaming. Click any [N] to jump to the source.
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
Scaling Security Alert Triage With Specialized Agents on Databricks
Databricks AI now enables automated, real-time triage of high-volume, low-severity security alerts using 17 specialized agents on Spark Structured Streaming. This approach achieved a 10x higher true-positive rate and saved over 6,500 analyst hours in the first month by ensuring every low-severity alert is investigated.
Delta Lake 4.3.0
Delta 4.3.0 deepens Unity Catalog integration by making it the source of truth for managed table operations via the UC Delta REST API, introduces replaceOn/replaceUsing DataFrame APIs for selective row-level data replacement, and improves UniForm with atomic Iceberg conversion and incremental metadata updates. Delta Sharing gains streaming support, Change Data Feed capabilities, and Trigger.AvailableNow, plus performance improvements like better V2 checkpoint parallelization and variant column statistics for data skipping.
TutorialsApache Spark Streaming Real-Time Mode - Latency Demo
The video demonstrates how to deploy and run Apache Spark Streaming in Real-Time Mode (RTM) using a declarative automation bundle. It shows that RTM significantly reduces P50 and P95 latencies compared to microbatch mode, achieving 26ms and 50ms respectively in a simplified setup without an external messaging bus.
TutorialsAir Traffic Control with Apache Spark Structured Streaming Real-Time Mode
The video demonstrates building a real-time air traffic control application using Apache Spark Structured Streaming Real-Time Mode, Lakehouse, and Databricks Apps. This system processes live flight telemetry, detects congestion, and generates alerts with sub-second end-to-end latency, all within a single Databricks platform.
TutorialsUnlock Your Use Cases: A Deep Dive on Structured Streaming’s New TransformWithState API
NewsCrypto at Scale: Building a High-Performance Platform for Real-Time Blockchain Data
NewsSupercharging Sales Intelligence: Processing Billions of Events via Structured Streaming
TutorialsReal-Time Mode Technical Deep Dive: How We Built Sub-300 Millisecond Streaming Into Apache Spark™
ReleasesNebula: The Journey of Scaling Instacart’s Data Pipelines with Apache Spark™ and Lakehouse
NewsUS Army Corp of Engineers Enhanced Commerce & National Sec Through Data-Driven Geospatial Insight
NewsHigh Volume Intelligent Streaming with Sub-Minute SLA for Near Real-Time Data Replication
NewsHow We Made a Unified Talent Solution Using Databricks Machine Learning, Fine-Tuned LLM & Dolly 2.0
Community
