Zerobus
Recent items mentioning Zerobus across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Practitioners are evaluating zerobus-ingest alongside Kafka 1 while navigating authentication setups involving credential selection 3 and oidc/v1/token REST endpoint configurations 2. Concurrently, architectures are expanding into IoT and sensor workloads through metadata-driven backdated extraction patterns 4 and real-time streaming demonstrations 5.
Generated daily from the 5 most recent items mentioning Zerobus. Click any [N] to jump to the source.
zerobus-ingest over kafka
I tested zerobus-ingest over the Kafka-compatible API. It worked fairly well but a little slower than I hoped. I'm guessing that is just the penalty that comes with using Kafka instead of the proprietary SDK. I can't use the proprietary SDK as of now, so I'm working within those constraints. I noticed that this Kafka interface seems to be restricted to json/text, which seems problematic where performance is concerned. Any thoughts on how to scale up and get more throughput? The docs clearly say that batching is the single biggest lever for throughput, but even after playing with batch sizes my data is still not uploading faster than 3 Mbps of json or so. I'm assuming there is no restriction (concurrency/blocking) that would prevent me from running ten kafka producers at a time? Maybe that allows me to get up to 30 Mbps. I'm really tempted to just go back to dropping parquet files in a temp folder. No matter what magic databricks has conjured in zerobus, I find it hard to believe they can compete with the performance of dropping blobs into storage. submitted by /u/SmallAd3697 [link] [comments]
Issues with oidc/v1/token endpoint (REST for zerobus-ingest)
Anyone else work with the zerobus-ingest? There is an auth-flow to get an "oauth token" from a service principal with an "oauth secret". In that flow, you are supposed to submit the authorization details. See docs: https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-kafka I had been requesting access (via authorization_details) to a table as shown in the example: "type": "unity_catalog_privileges", "privileges": ["SELECT"], "object_type": "TABLE", "object_full_path": "main.default.air_quality" ... but it wouldn't work. Was banging my head on it all day. The privileges granted to the service principal for this table were unrestricted " ALL PRIVILEGES ". It turns out that, in addition to "ALL PRIVILEGES", this "token" endpoint wanted me to grant "SELECT " to the service principal in UC governance as well. Else the authorization_details are not valid. This "token" api has a lot of sharp edges. Any chance there is a nuget client to wrap around it and protect us from these unfortunate nuances? Is it documented that ALL PRIVILEGES doesn't actually give us all privileges, if we request "SELECT" by name? submitted by /u/SmallAd3697 [link] [comments]
What credential should I use to authenticate Databricks Zerobus?
I’m learning how to use Databricks Zerobus and I’m confused about authentication. When connecting to Zerobus, should I use a Databricks personal access token, OAuth credentials, a service principal, or another type of credential? I’m testing this for a learning project and want to follow the recommended secure approach. I won’t post any actual credentials. What authentication method are people using, and are there any setup steps or permissions I should know about? submitted by /u/Stu-dent999 [link] [comments]
A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks
Community BrickTalk | Real-Time Data & AI: Tripwise Demo
Hey r/Databricks ! Join us for community BrickTalk on Thursday, September 24 , focusing on real-time data streaming, AI agents, and governance using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Data and AI, giving customers a direct line to the people behind the products. In this session, we'll walk through a live demonstration of the Tripwise Demo , featuring: Sub-Second Transactions & Streaming: Device registration into Lakebase with sub-second reads/writes, plus telemetry streaming via Zerobus through a governed Medallion architecture. AI-Generated Offers & Pricing: Generating real-time agent offers using Foundation Model APIs and scoring behavioral data for usage-based renewal pricing. Natural Language Analytics: Enabling underwriters, product managers, and marketing teams to query governed insurance data in seconds using AI/BI Dashboards and Genie. Unified Governance: Managing safety, compliance, and control end-to-end with Unity Catalog. This is a great chance to see real-world architecture in action and ask questions directly to Databricks experts. When: Thursday, September 24 9:00 AM PT 12:00 PM ET 5:00 PM BST (London) 9:30 PM IST Register here and save your spot submitted by /u/Subject_Ant1789 [link] [comments]
Databricks zerobus vs Fabric open mirroring (2026)
Has anyone seen any comparison between the generalized ingestion mechanism (zerobus) with Fabric's offering (open mirroring)? Seems like there should be a blog or youtube video comparing the two by now. But I haven't seen any. Unfortunately it sounds like they both rely on proprietary middleware. Ideally there would be a similar type of software which that we could just run on-premise to land data into cloud blobs (like a gateway of some kind). Not sure why that would be so hard for someone to do as a github library or something. Maybe it would need to be done in a performant language like rust or .net, but it doesn't seem like it would be rocket science. Both those technologies are relatively recent: Fabric open mirroring : May 2025 Zerobus : Feb 2026 Personally I wouldn't want to pick either one of these technologies until a comparison could be made. Microsoft's open mirroring claims that they can land data in their lakehouses for free. After that point, the raw deltalake tables would be accessible to both platforms. If open mirroring is truly free then it seems odd that any databricks customers would be using zerobus. They should just purchase the smallest possible capacity from Microsoft like an F2, and use that for moving all their data to raw/bronze in adls gen2 containers. Whatever happens after that can take place in either of these two saas'es, databricks or fabric. submitted by /u/SmallAd3697 [link] [comments]
CUSTOMER STORY | From high costs to near real-time insights with Zerobus Ingest
Plug & Play: Zerobus Ingest Now Supports Apache Kafka® Compatible APIs (Beta)
Try the new Zerobus Rescue Column: Choose-Your-Own-Adventure Schema Management (Beta)
Agents for production lines: Trusted decisions in real time
Agents like ProdLine CoPilot analyze live plant state from the Databricks Data Intelligence Platform to turn line-manager recovery decisions from a next-morning report into an in-shift answer. Zerobus streams OT telemetry into Delta tables alongside MES, ERP, and LIMS under Unity Catalog, so domain specialists can run real solvers against planners' existing constraints while recommendations stay as draft work orders and holds pending human approval.
NewsLearn about Zerobus in 15 min!
Databricks Lakeflow Connect Zerobus Ingest is a high-performance, multi-cloud ingestion service that allows users to stream event data directly into their lakehouse without the cost and complexity of a traditional message bus. The video explains the architecture of Zerobus Ingest, announces upcoming API integrations for Kafka and MQTT, and demonstrates how to configure and run a Python client to write data directly into a Delta table.
Zerobus ingestion fails in Databricks
Zerobus ingestion fails in Databricks Free Edition using Service Principal OAuth
Real Time Healthcare Analytics with Databricks Zerobus and Lakehouse RT via Reyden
EventsUnlocking agentic data engineering with Lakeflow + Genie
The video introduces Lakeflow as a unified, open data engineering stack that simplifies data transformation, ingestion, and orchestration through declarative pipelines, no-code tools, and managed services. It also announces Genie Zero Ops, an AI agent that automates data operations by autonomously detecting, diagnosing, and verifying fixes for data incidents and PII exposures within the data plane.
Lakeflow: A new era of agentic data engineering
Lakeflow unifies ingestion, transformation, and orchestration under Unity Catalog, providing a single source of trusted, real-time context for agentic AI. It offers high-performance ingestion from 100+ sources, real-time streaming, visual pipeline building with Lakeflow Designer, and AI-powered authoring and operations with Genie Code and Genie ZeroOps.
Petabyte scale with Zerobus ingest: Download the Code and Ingest the Milky Way
Ingesting the Milky Way: Petabyte-Scale with Zerobus Ingest
Zerobus Ingest, a new serverless streaming API, enables instant deployment of petabyte-scale data pipelines on Databricks without manual infrastructure management. Its dynamic partitioning architecture automatically scales compute and sustains over 12 GB/s throughput to a single table, efficiently handling unpredictable data volumes.
Databricks' Zerobus Event Data Ingestion Deep-Dive Demo (w/ Databricks' Staff Developer)
So very excited to share this demo + presentation with the one and only, [Scott Haines](https://www.linkedin.com/feed#), Staff Developer Advocate @ [Databricks](https://www.linkedin.com/feed#). The topic? Zerobus, which is a great option for easily ingesting event data at scale into Unity Catalog. We do a demo and overview of the technology, talk about how it is similar & different to Kafka, when to use Real-Time Mode vs Zerobus, and much more! Hope you enjoy this very technical overview!
NewsZerobus Ingest, Lakebase and Databricks Apps in Action: Data Streaming with Databricks
The video demonstrates a real-time IoT data streaming application built with Zerobus for ingestion, Lakebase for low-latency serving, and Databricks Apps for the front and back ends. This architecture processes thousands of concurrent IoT events from mobile phone sensors globally without using Kafka or traditional complex pipelines.
From months to minutes: Building real-time clinical data pipelines with natural language
Databricks and Redox now enable real-time clinical data pipelines from EHRs to Unity Catalog with natural language prompts, reducing integration time from months to minutes. This partnership allows AI outputs to be written back into the EHR in real time, transforming Databricks into an operational layer for point-of-care interventions.
CUSTOMER STORY | Toyota uses Zerobus Ingest for real-time factory data
NewsZerobus Ingest and Lakebase in Action: Data Streaming with Databricks
The video demonstrates a real-time IoT data streaming application built with Zerobus for ingestion, Lakebase for low-latency serving, and Databricks apps for the front and back end, without relying on Kafka. It showcases how thousands of concurrent IoT events from mobile phone sensors worldwide are ingested, processed, and visualized on a map, with traces served by Lakebase for fast access.
Tutorials54 Zerobus Ingest Lakeflow Standard Connector | Ingest Streaming data directly into Delta Table
The video demonstrates how to use Databricks Zero Bus Ingest, a push-based API, to directly stream various data types like IoT, event, and telemetry data into Unity Catalog Delta tables. It highlights Zero Bus Ingest's ability to simplify streaming ingestion by eliminating the need for intermediate message buses and managing their infrastructure.
EventsData + AI Summit Keynote Day 2
Databricks announced major expansions to its data platform including managed Apache Iceberg tables in Unity Catalog with governance across multiple engines, and LakeFlow Designer for building production ETL pipelines without code. The company also contributed real-time streaming mode and declarative pipelines to Apache Spark, and launched Zerobus, a direct write API enabling terabytes-per-hour data ingestion into the lakehouse.
Get Tuesday's version of this
Tracking Zerobus? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.