Skip to content
All videos
newsDatabricks·October 21, 2019

Improving Apache Spark Downscaling - Christopher Crosbie (Google) Ben Sidhom (Google)

Summary

Apache Spark downscaling becomes critical in serverless-like environments and with preemptible or spot instances, where frequent node removal can trigger repeated expensive recomputation. Google's Christopher Crosbie and Ben Sidhom explore approaches to improve scale-down on open source cluster managers such as YARN and Kubernetes, from job scheduling to block and shuffle placement.

Summary generated by brickster.ai from the video transcript.

More from Databricks