After analyzing the performance of several common lossless compression algorithms, Dropbox engineers have slightly modified Google’s Brotli encoder to improve their engine sync performance. They could thus reduce median latency and data transfer by more than 30%, Dropbox engineers Rishabh Jain and Daniel Reiter Horn maintain.

Dropbox has been using compression techniques to improve syncing performance since they wrote their original sync engine. However, recent advancements in the field of lossless compression motivated Dropbox engineers to explore the possibility of moving away from the old zlib algorithm they had originally adopted.

Broccoli is essentially an extension of Brotli which is able to concatenate Brotli files with one another. This enabled higher compression rates of up to 3x the rate of Brotli original implementation thanks to the use of multiples cores to compress chunks of a file and then concatenate them together. Jain and Reiter Horn enumerate a number of key advantages Brotli had in their view, including the possibility of dynamically selecting Huffman tables at runtime, a safe, Rust-based implementation, and being standard for HTTP.

#performance & scalability #architecture #compression #rust #cloud #architecture & design #development #news