lzbench VS lizard

For a benchmark on a standard set: https://github.com/inikep/lzbench/blob/master/lzbench18_sort...

Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench

You can crunch the numbers yourself with this: https://github.com/inikep/lzbench

Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1], which is more or less up-to-date.

[1] https://github.com/inikep/lzbench

u/zertillon, maybe you could use lzbench, so you could compare it with a lot of other compression libraries. The problem is that it requires including the library in a single executable, so it might be more difficult to integrate than a C library (the benchmark is in C++).

Still updated: https://github.com/inikep/lzbench

If you want to know the size a VM will compress to,. Zstd can be installed on any machine, so you can experiment easily. You can even run the benchmark https://github.com/inikep/lzbench

Are you sure about that? That's not what I see on https://github.com/inikep/lzbench and I tried to run that myself, although I have no idea which lzo to try so I went with what seemed the fastest...

Another fun trick is to compress less often used stuff in RAM memory, because decompressing something like LZ4 or Lizard is still potentially orders of magnitude faster than reading from disk.

Lizard - Formerly LZ5; an efficient compressor with fast decompression. Achieves compression ratios comparable with zip and zlib at decompression speeds of 1000MB/s and faster. BSD-2-Clause