zstd VS zlib-ng

If zstd give you an error and you don't handle it, the next calls may cause UB, so it kinda does both things.

https://github.com/facebook/zstd/blob/b16d193512d3ded82fd584...

( https://github.com/facebook/zstd/releases/tag/v1.5.6 )

In my opinion, it is better to check the original repository: https://github.com/inikep/lzbench

This. Even their less known work is pretty solid[1] ( used it the other day and was frankly kinda amazed at how well it performed under the circumstances ). Facebook/Meta sucks like most social madia does, but, not unlike Elon Musk, they are on the record of having some contributions to society as a whole.

[1]https://github.com/facebook/zstd

I don't think so? It's only seekable with an additional index [1], just like any other compression scheme.

[1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

- latest zstd v1.5.6 ( Mar 30, 2024 https://github.com/facebook/zstd/releases )

You may have also noticed that a new supported data compression format zstd appeared some time ago. I haven't seen any backends that use it yet, but httpx will support decompression in versions above 0.28.0. I already use it to compress server response dumps in my projects; it shows incredible efficiency in asynchronous solutions with aiofiles.

Zstd is a personal project? Surely it's not by accident in the Facebook GitHub organization? And that you need to sign a contract on code.facebook.com before they'll consider merging any contributions? That seems like an odd claim, unless it used to be a personal project and Facebook took it over

(https://github.com/facebook/zstd/blob/dev/CONTRIBUTING.md#co...)

Unmount root and remount the subvolumes and the boot partition. noatime is used for better performance zstd as file compression:

I'm not sure why people say this about certain languages (it is sometimes said about Haskell, as well).

The code has a C style to it, but that doesn't mean it wasn't actually written in Rust -- Rust deliberately has features to support writing this kind of code, in concert with safer, stricter code. This isn't bad, it's good. Imagine if we applied this standard to C code. "Zlib-NG is basically written in assembler, not C..." https://github.com/zlib-ng/zlib-ng/blob/50e9ca06e29867a9014e...

Looks like it depends on https://github.com/nodeca/pako for the zlib compression.

> Almost as fast in modern JS engines as C implementation (see benchmarks).

Impressive, although zlib itself is no longer the bar to beat for zlib, I think that goes to https://github.com/zlib-ng/zlib-ng these days

For what it’s worth, the benchmark on the Zstandard homepage[1] shows none of the compressors tested breaking 1GB/s on compression, and only the fastest and sloppiest ones breaking 1GB/s on decompression. If you’re OK with its API limitations, libdeflate[2] is known to squeeze past 1GB/s decompressing normal Deflate compression levels. So asking for multiple GB/s is probably unfair.

Still, 10MB/s sounds like the absolute minimum reasonable speed, and they’re reporting nearly three orders of magnitude below that. A modern compressor does not run at bad dialup speeds; something in there is absolutely murdering the performance.

And it might just be the constant-time overhead, as far as I can see. The article mentions “a few hundred bytes” of payload, and the discussion of measurements implies 1.5KB uncompressed. Even though they don’t reinitialize the compressor on each message, that is still a very very modest amount of data.

So it might be that general-purpose compressors are just a bad tool here from a performance standpoint. I’m not aware of a good tool for this kind of application, though.

[1] https://facebook.github.io/zstd/#benchmarks

[2] https://github.com/zlib-ng/zlib-ng/issues/1486

Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:

https://github.com/zlib-ng/zlib-ng/discussions/871

IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.

It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.

Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328

Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.

I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.

https://github.com/zlib-ng/zlib-ng/discussions/871

zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng

Might be interesting to benchmark their implementation too to see how it compares.