zlib-ng VS libjpeg-turbo

Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:

https://github.com/zlib-ng/zlib-ng/discussions/871

IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.

It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.

Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328

Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.

I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.

https://github.com/zlib-ng/zlib-ng/discussions/871

zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng

Might be interesting to benchmark their implementation too to see how it compares.

zlib-ng: https://github.com/zlib-ng/zlib-ng/blob/develop/functable.c

(2) There are many packages that rely upon zlib and minizip and switching those underlying dependencies is easier said than done. We can't drop zlib completely and switch: "The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change." - https://github.com/zlib-ng/zlib-ng

There are already active zlib forks (e.g. https://github.com/zlib-ng/zlib-ng), the problem is with having people move to them. It takes a lot of effort to move mindshare from the original version to a fork, there's some historical examples of it happening, but not a ton.

> all decoders will render the same pixels

Not true. Even just within libjpeg, there are three different IDCT implementations (jidctflt.c, jidctfst.c, jidctint.c) and they produce different pixels (it's a classic speed vs quality trade-off). It's spec-compliant to choose any of those.

A few years ago, in libjpeg-turbo, they changed the smoothing kernel used for decoding (incomplete) progressive JPEGs, from a 3x3 window to 5x5. This meant the decoder produced different pixels, but again, that's still valid:

https://github.com/libjpeg-turbo/libjpeg-turbo/commit/6d91e9...

Last vestiges of this fact AFAIK were libjpeg, which had a macro NEED_SHORT_EXTERNAL_NAMES that shortens all public identifiers to have unique 6-letter-long prefixes. Libjpeg-turbo nowadays has removed them though [1].

[1] https://github.com/libjpeg-turbo/libjpeg-turbo/commit/52ded8...

While I think the move to safer code through Rust and other alternatives is a nice breath of fresh air, I doubt you can get these kinds of optimization without using unsafe code in Rust. These optimized implementations often require some kind of safety-bypassing memory modifications to work as efficiently ad they do.

There's a reason https://github.com/libjpeg-turbo/libjpeg-turbo/tree/main/sim... is filled with assembly files with conditional loading.

Sure. You'll see it very often in codec implementations. From rav1e, a fast AV1 encoder mostly written in Rust: https://github.com/xiph/rav1e/tree/master/src/x86

Large portions of the algorithm have been translated into assembly for ARM and x86. Shaving even a couple percent off something like motion compensation search will add up to meaningful gains.

Or the current reference implementation of JPEG: https://github.com/libjpeg-turbo/libjpeg-turbo/tree/main/sim...

zune-jpeg is 1.5x to 2x faster than jpeg-decoder and is on par with libjpeg-turbo.

This is interesting from the standpoint of how new JVM features can be used to improve performance (what I presume the article's main purpose to have been), but the image processing improvement itself isn't head-turning. Also, we've found that libjpeg-turbo (https://libjpeg-turbo.org/) is ~5x (IIRC, can re-run my JMH benchmark if anyone wants me to) as fast for decoding JPEGs as ImageIO, so we wouldn't even benefit from this change in 21 much.

libjpeg-turbo: https://github.com/libjpeg-turbo/libjpeg-turbo/blob/main/simd/x86_64/jsimdcpu.asm

libjpeg-turbo

For all but the simplest formats (basically BMP), the difficulty of implementing encoding/decoding from scratch is significant - well beyond a beginner's ability, and challenging/time-consuming even for senior developers. So, libraries are used in practice - e.g. libpng and libjpeg-turbo.