Skip to content
All videos
newsDatabricks·September 17, 2021

The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro

Summary

Zstandard, a fast compression algorithm, is traced through four Apache Spark use cases: shuffle file compression to cut local disk IO on K8s, event log compression where it replaced LZ4 as default, ORC and Parquet data files, and MapStatus serialization. The talk also looks at Avro support targeted for Spark 3.2.

Summary generated by brickster.ai from the video transcript.

More from Databricks