SIMDCompressionAndIntersection VS lzbench

If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1].

Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory.

That said, I think your first point is still reasonable for certain types of data and compression algorithms.

[1] https://github.com/lemire/SIMDCompressionAndIntersection

For a benchmark on a standard set: https://github.com/inikep/lzbench/blob/master/lzbench18_sort...

Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench

You can crunch the numbers yourself with this: https://github.com/inikep/lzbench

Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1], which is more or less up-to-date.

[1] https://github.com/inikep/lzbench

u/zertillon, maybe you could use lzbench, so you could compare it with a lot of other compression libraries. The problem is that it requires including the library in a single executable, so it might be more difficult to integrate than a C library (the benchmark is in C++).

Still updated: https://github.com/inikep/lzbench

If you want to know the size a VM will compress to,. Zstd can be installed on any machine, so you can experiment easily. You can even run the benchmark https://github.com/inikep/lzbench

Are you sure about that? That's not what I see on https://github.com/inikep/lzbench and I tried to run that myself, although I have no idea which lzo to try so I went with what seemed the fastest...