Skip to content
All videos
newsDatabricks·August 31, 2021

Hybrid Apache Spark Architecture with YARN and Kubernetes

Summary

Lyft runs ETL and ML Spark workloads on Kubernetes with the Spark on k8s operator, but growing scale caused reliability issues with IP allocation, container images, IAM roles and the control plane. A hybrid architecture on Kubernetes and YARN supports continued growth, with a dynamic runtime controller for per-environment config overrides and easy switchover.

Summary generated by brickster.ai from the video transcript.

More from Databricks