Skip to content
All topics

Zerobus

Recent items mentioning Zerobus across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

41 recent items5 news7 videos29 community threads
What's happening in ZerobusAI synthesis · updated Nov 16, 2026

Push-based change data capture can now stream directly into Unity Catalog Delta tables using Debezium Server and Zerobus Ingest, eliminating the need for intermediate Kafka brokers or pull-based connectors 56. This outbound architecture allows locked-down private networks to replicate directly to the lakehouse 6, while practitioners extend the technology to support private-endpoint MLflow traces 9 and sub-second operational workloads alongside Lakehouse//RT 10.

Generated daily from the 10 most recent items mentioning Zerobus. Click any [N] to jump to the source.

Databricks CommunityData Engineering

Zerobus Ingest availability in Azure South Africa North (southafricanorth)?

002d ago
Databricks CommunityData Engineering

Zerobus to lakebase

001w ago
Databricks CommunityBrickTalks TV

Recording | Bricktalk: Zerobus Ingest & CDC Workflows

001w ago
Reddit

Structured Streaming Statestore is Not Working (Zerobus)

Zerobus is writing data to a landing table like so: CREATE TABLE a.b.streaming_c ( MyA STRING NOT NULL, MyB INT NOT NULL, MyC STRING, MyDetails VARIANT, ZerobusRescueJson VARIANT) USING delta TBLPROPERTIES ( 'delta.checkpoint.writeStatsAsJson' = 'false', 'delta.checkpoint.writeStatsAsStruct' = 'true', 'delta.enableDeletionVectors' = 'true', 'delta.enableRowTracking' = 'true', 'delta.feature.appendOnly' = 'supported', 'delta.feature.deletionVectors' = 'supported', 'delta.feature.domainMetadata' = 'supported', 'delta.feature.invariants' = 'supported', 'delta.feature.rowTracking' = 'supported', 'delta.feature.variantType' = 'supported', 'delta.minReaderVersion' = '3', 'delta.minWriterVersion' = '7', 'delta.parquet.compression.codec' = 'zstd', 'delta.parquet.format.version' = '2.12.0', 'delta.parquet.format.version.afe.internal' = '2.12.0', 'delta.rowTracking.materializedRowCommitVersionColumnName' = '_row-commit-version-col-777ca68b-c04b-4b5d-9a9d-5c5535e32d20', 'delta.rowTracking.materializedRowIdColumnName' = '_row-id-col-1885c4cd-1354-4ce6-9726-ed098a251a62', 'io.unitycatalog.tableId' = '6f07a65b-4887-4aa2-8f56-569b2922aec4') I use readStream/writeStream to move the data to a different place. I set the checkpointLocation (ie. .option(" checkpointLocation ", x)) to an abfss location in azure: "abfss:// datalake@myctr01.dfs.core.windows.net /test002/x05" https://preview.redd.it/9e8bc5s9pvzh1.png?width=653&format=png&auto=webp&s=84b3770e88b60503698db3831ea020dfe07add8e Later I try to audit the state of the checkpointLocation and it fails: SELECT * FROM read_statestore('abfss://datalake@abc.dfs.core.windows.net/test002/x05/') [STDS_FAILED_TO_READ_STATE_SCHEMA] Failed to read the state schema. Either the file does not exist, or the file is corrupted. options: StateSourceOptions(checkpointLocation=abfss:// datalake@xxx.dfs.core.windows.net /test002/x05, batchId=1, operatorId=0, storeName=default, joinSide=none, stateVarName=None, +flattenCollectionTypes=true). Rerun the streaming query to construct the state schema, and report to the corresponding communities or vendors if the error persists. SQLSTATE: 42K03 That checkpointLocation is very intimidating to me. It is doing ALL the heavy lifting when it comes to keeping track of the zerobus records that have been processed. There needs to be a way to recover an audit trail from this statestore location. Why is the statestore corrupted? I tried the pyspark equivalent of this SELECT statement and it has the same error. What is more frustrating is how little information I can find about this statestore, despite the fact that the tech is ten years old. How are others accomplishing this goal? There needs to be a simple approach for auditing the underlying state that is managed by spark structured streaming. I'm using dbx 16.4 and spark 3.5.2 submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36971w ago
Databricks CommunityCommunity Articles

Zerobus Ingest hands-on: Postgres CDC to Delta with Debezium Server, no Kafka

001w ago
Reddit

Template columns for zerobus-ingest

I'm pretty certain this would go way down at the bottom of the priority list. It may never be implemented at all... But I kind of wish that zerobus would have a way to fill template fields (for self-determined details about the data, like the exact UTC time a message was made durable, and the exact time it was queryable, and some sort of an incrementing sequence number, and maybe a transaction ID of some kind). Some of these things can be implemented in custom code inside the client. However some of them are not known to the client. Either way, it would be better if the zerobus itself could fill these types of general-purpose template fields. Perhaps those fields could be dropped into a single json column, similar to the zerobus-rescue. Zerobus-ingest hasn't been GA for very long, and maybe these sorts of features will be added down the road. I thought I'd at least put a placeholder here, to start the discussion. submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36972w ago
Reddit

Bricktalk: Zerobus Ingest & CDC Workflows

Hello Zerobus Ingest friends! Mark your calendars for the Zerobus BricksTalk featuring Omkar Mehta from NVIDIA and Wagner Santos , about their contribution of Zerobus as a sink in Debezium. We will discuss why they are using Debezium for Postgres CDC ingestion and how Zerobus is strategically important for streamlining data to Databricks. We will cover: - Debezium Integration: Setting up CDC ingestion through Debezium directly to Zerobus Ingest. - Bypassing Kafka: Leveraging Zerobus as a destination to streamline lakehouse data ingestion without needing dedicated intermediate message brokers. - Product Vision and Roadmap: An overview of Zerobus capabilities and upcoming features from Databricks product leadership. Date and Time: Thursday, November 5, 2026 PT: 9:00 AM ET: 12:00 PM London (GMT): 5:00 PM IST: 10:30 PM 👉 Register today to save your spot! https://usergroups.databricks.com/events/details/databricks-user-groups-bricktalks-presents-community-bricktalk-zerobus-ingest-amp-cdc-workflows/ https://i.redd.it/mdkiptssofzh1.gif submitted by /u/Defiant-Pause9053 [link] [comments]

00Defiant-Pause90532w ago
Databricks CommunityData Engineering

Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)

002w ago
Databricks CommunityTechnical Blog

End-to-End Cyber Lakehouse Streaming via Zerobus Ingest, Spark Declarative Pipelines, and OCSF

003w ago
Databricks CommunityAnnouncements

Upcoming Community BrickTalk | Zerobus Ingest & CDC Workflows

003w ago
Databricks CommunityData Engineeringanswered

Zerobus Variant Data not Processed as Expected (json data destined for Variant column).

001mo ago
Databricks CommunityData Engineeringanswered

Zerobus rescue (VARIANT) not working. No Preview Available.

001mo ago
Reddit

Zerobus message retrieval

I can see how zerobus is used for streaming data into a lakehouse table. I can generally see how you would consume the data as well. A couple things are still not clear. What is the pattern for removing old rows out of the UC table? Should we do a delete once a year, on messages older than 3 years? Is this a necessary maintenance that we are responsible for, or is there something in the software which will help with housekeeping? And how would I perform scheduled tasks against this data? Let's say that data represents a news feed and I just need to send an summarized overview once an hour, and then flag the rows that have been summarized. I'm assuming it would be disruptive to try to delete the rows that were processed or even update them (ie. it may interfere with zerobus itself, and create concurrency conflicts). Should I build a DIFFERENT table that tracks the surrogate IDs of rows already processed/summarized? Sorry for the basic questions, but I wasn't able to find the answer to these yet. I'm sure I could find my own solution, but it would be interesting to hear if there is a common practice submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36971mo ago
Databricks CommunityDatabricks TV

Learn about Zerobus in 15 min!

001mo ago
Databricks CommunityGet Started Discussions

Bypassing the Message Broker with Databricks Zerobus Ingest

001mo ago
Databricks CommunityData Engineering

Zerobus Ingest with Kafka compliant API and WITHOUT a callback

001mo ago
Reddit

zerobus-ingest over kafka

I tested zerobus-ingest over the Kafka-compatible API. It worked fairly well but a little slower than I hoped. I'm guessing that is just the penalty that comes with using Kafka instead of the proprietary SDK. I can't use the proprietary SDK as of now, so I'm working within those constraints. I noticed that this Kafka interface seems to be restricted to json/text, which seems problematic where performance is concerned. Any thoughts on how to scale up and get more throughput? The docs clearly say that batching is the single biggest lever for throughput, but even after playing with batch sizes my data is still not uploading faster than 3 Mbps of json or so. I'm assuming there is no restriction (concurrency/blocking) that would prevent me from running ten kafka producers at a time? Maybe that allows me to get up to 30 Mbps. I'm really tempted to just go back to dropping parquet files in a temp folder. No matter what magic databricks has conjured in zerobus, I find it hard to believe they can compete with the performance of dropping blobs into storage. submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36971mo ago
Reddit

Issues with oidc/v1/token endpoint (REST for zerobus-ingest)

Anyone else work with the zerobus-ingest? There is an auth-flow to get an "oauth token" from a service principal with an "oauth secret". In that flow, you are supposed to submit the authorization details. See docs: https://learn.microsoft.com/en-us/azure/databricks/ingestion/zerobus-kafka I had been requesting access (via authorization_details) to a table as shown in the example: "type": "unity_catalog_privileges", "privileges": ["SELECT"], "object_type": "TABLE", "object_full_path": "main.default.air_quality" ... but it wouldn't work. Was banging my head on it all day. The privileges granted to the service principal for this table were unrestricted " ALL PRIVILEGES ". It turns out that, in addition to "ALL PRIVILEGES", this "token" endpoint wanted me to grant "SELECT " to the service principal in UC governance as well. Else the authorization_details are not valid. This "token" api has a lot of sharp edges. Any chance there is a nuget client to wrap around it and protect us from these unfortunate nuances? Is it documented that ALL PRIVILEGES doesn't actually give us all privileges, if we request "SELECT" by name? submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36971mo ago
Reddit

What credential should I use to authenticate Databricks Zerobus?

I’m learning how to use Databricks Zerobus and I’m confused about authentication. When connecting to Zerobus, should I use a Databricks personal access token, OAuth credentials, a service principal, or another type of credential? I’m testing this for a learning project and want to follow the recommended secure approach. I won’t post any actual credentials. What authentication method are people using, and are there any setup steps or permissions I should know about? submitted by /u/Stu-dent999 [link] [comments]

00Stu-dent9992mo ago
Databricks CommunityCommunity Articles

A Metadata-Driven Backdated Extraction Pattern for IoT/Sensor Pipelines Using Zerobus and Databricks

002mo ago
Reddit

Community BrickTalk | Real-Time Data & AI: Tripwise Demo

Hey r/Databricks ! Join us for community BrickTalk on Thursday, September 24 , focusing on real-time data streaming, AI agents, and governance using Databricks. BrickTalks is a community event series where Databricks experts share real-world use cases, demos, and practical insights for building with Data and AI, giving customers a direct line to the people behind the products. In this session, we'll walk through a live demonstration of the Tripwise Demo , featuring: Sub-Second Transactions & Streaming: Device registration into Lakebase with sub-second reads/writes, plus telemetry streaming via Zerobus through a governed Medallion architecture. AI-Generated Offers & Pricing: Generating real-time agent offers using Foundation Model APIs and scoring behavioral data for usage-based renewal pricing. Natural Language Analytics: Enabling underwriters, product managers, and marketing teams to query governed insurance data in seconds using AI/BI Dashboards and Genie. Unified Governance: Managing safety, compliance, and control end-to-end with Unity Catalog. This is a great chance to see real-world architecture in action and ask questions directly to Databricks experts. When: Thursday, September 24 9:00 AM PT 12:00 PM ET 5:00 PM BST (London) 9:30 PM IST Register here and save your spot submitted by /u/Subject_Ant1789 [link] [comments]

00Subject_Ant17892mo ago
Reddit

Databricks zerobus vs Fabric open mirroring (2026)

Has anyone seen any comparison between the generalized ingestion mechanism (zerobus) with Fabric's offering (open mirroring)? Seems like there should be a blog or youtube video comparing the two by now. But I haven't seen any. Unfortunately it sounds like they both rely on proprietary middleware. Ideally there would be a similar type of software which that we could just run on-premise to land data into cloud blobs (like a gateway of some kind). Not sure why that would be so hard for someone to do as a github library or something. Maybe it would need to be done in a performant language like rust or .net, but it doesn't seem like it would be rocket science. Both those technologies are relatively recent: Fabric open mirroring : May 2025 Zerobus : Feb 2026 Personally I wouldn't want to pick either one of these technologies until a comparison could be made. Microsoft's open mirroring claims that they can land data in their lakehouses for free. After that point, the raw deltalake tables would be accessible to both platforms. If open mirroring is truly free then it seems odd that any databricks customers would be using zerobus. They should just purchase the smallest possible capacity from Microsoft like an F2, and use that for moving all their data to raw/bronze in adls gen2 containers. Whatever happens after that can take place in either of these two saas'es, databricks or fabric. submitted by /u/SmallAd3697 [link] [comments]

00SmallAd36972mo ago
Databricks CommunityAnnouncements

CUSTOMER STORY | From high costs to near real-time insights with Zerobus Ingest

003mo ago
Databricks CommunityTechnical Blog

Plug & Play: Zerobus Ingest Now Supports Apache Kafka® Compatible APIs (Beta)

003mo ago
Databricks CommunityTechnical Blog

Try the new Zerobus Rescue Column: Choose-Your-Own-Adventure Schema Management (Beta)

003mo ago
Databricks CommunityAdministration & Architecture

Zerobus ingestion fails in Databricks

004mo ago
Databricks CommunityAdministration & Architectureanswered

Zerobus ingestion fails in Databricks Free Edition using Service Principal OAuth

004mo ago
Databricks CommunityCommunity Articles

Real Time Healthcare Analytics with Databricks Zerobus and Lakehouse RT via Reyden

004mo ago
Databricks CommunityTechnical Blog

Petabyte scale with Zerobus ingest: Download the Code and Ingest the Milky Way

005mo ago

Get Tuesday's version of this

Tracking Zerobus? The Tuesday email carries what moved across the whole ecosystem, not just this topic. Free, one-click unsubscribe.

Read past issues first