Skip to content
All videos
newsDatabricks·July 19, 2022

How socat and UNIX Pipes Can Help Data Integration

Summary

Containerized CLIs, one reading and one writing data, are reliably orchestrated on Kubernetes for data integration through a TCP-based process abstraction built with socat and UNIX pipes. Architectures for horizontally scaling these workflows are compared, and the approach powers all of Airbyte's Kubernetes deployments, syncing TBs of data daily.

Summary generated by brickster.ai from the video transcript.

Topics

More from Databricks