precomp-cpp VS zlib-ng

You should look at Precomp - an apparently abandoned, but awesomely working arhiver that is dedicated to compressing those ZIP-based formats, and compresses them twice-thrice as small as LZMA2 can.

You can also have a look at this packer. It compresses PDF and EPUB 2-3 times smaller than 7z at maximum settings, at half the speed. I keep all my books in it and never had a problem.

There is an opensource archiver called precomp that is designed specifically for e-documents and compresses them 2-3 times smaller than 7z in ultra mode. It works, and I keep my e-books in it. However, it seems nobody else does, and development has stopped due to lack of interest.

There are tools like preflate [1] or precomp [2] that guarantees a bitwise identical reconstruction, of course modulo bugs.

[1] https://github.com/deus-libri/preflate

[2] https://github.com/schnaader/precomp-cpp/ (which internally makes use of preflate)

You may use "precomp" in order to seek and uncompress those parts. Then you can use a better compression algorithm : http://schnaader.info/precomp.php

Precomp http://schnaader.info/precomp.php is a program which will seek compressed parts and decompress those parts in order to compress with another algorithm. This greatly improve compression ratio with the other algorithm.

This may be of help. (a255854 and d649919)

Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:

https://github.com/zlib-ng/zlib-ng/discussions/871

IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.

It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.

Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328

Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.

I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.

https://github.com/zlib-ng/zlib-ng/discussions/871

zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng

Might be interesting to benchmark their implementation too to see how it compares.

zlib-ng: https://github.com/zlib-ng/zlib-ng/blob/develop/functable.c

(2) There are many packages that rely upon zlib and minizip and switching those underlying dependencies is easier said than done. We can't drop zlib completely and switch: "The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change." - https://github.com/zlib-ng/zlib-ng

There are already active zlib forks (e.g. https://github.com/zlib-ng/zlib-ng), the problem is with having people move to them. It takes a lot of effort to move mindshare from the original version to a fork, there's some historical examples of it happening, but not a ton.