python-zstandard VS lzbench

I'm working on file formats called tif file which are on average a gigabyte of size. A middle step is to compress those files and I looked it up and found this and tried to use it. But unfortunately, the size of the compressed file is barely 100MB less than the original one and it takes like 1.5 minutes to compress it. I'm reading the entire file in memory instead of streaming it but I don't really think that could result in compression size. I'm also using high enough level of compression(20 level) but it still won't give me good results.

For a benchmark on a standard set: https://github.com/inikep/lzbench/blob/master/lzbench18_sort...

Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench

You can crunch the numbers yourself with this: https://github.com/inikep/lzbench

Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1], which is more or less up-to-date.

[1] https://github.com/inikep/lzbench

u/zertillon, maybe you could use lzbench, so you could compare it with a lot of other compression libraries. The problem is that it requires including the library in a single executable, so it might be more difficult to integrate than a C library (the benchmark is in C++).

Still updated: https://github.com/inikep/lzbench

If you want to know the size a VM will compress to,. Zstd can be installed on any machine, so you can experiment easily. You can even run the benchmark https://github.com/inikep/lzbench

Are you sure about that? That's not what I see on https://github.com/inikep/lzbench and I tried to run that myself, although I have no idea which lzo to try so I went with what seemed the fastest...