C zlib-replacement Projects
-
For what it’s worth, the benchmark on the Zstandard homepage[1] shows none of the compressors tested breaking 1GB/s on compression, and only the fastest and sloppiest ones breaking 1GB/s on decompression. If you’re OK with its API limitations, libdeflate[2] is known to squeeze past 1GB/s decompressing normal Deflate compression levels. So asking for multiple GB/s is probably unfair.
Still, 10MB/s sounds like the absolute minimum reasonable speed, and they’re reporting nearly three orders of magnitude below that. A modern compressor does not run at bad dialup speeds; something in there is absolutely murdering the performance.
And it might just be the constant-time overhead, as far as I can see. The article mentions “a few hundred bytes” of payload, and the discussion of measurements implies 1.5KB uncompressed. Even though they don’t reinitialize the compressor on each message, that is still a very very modest amount of data.
So it might be that general-purpose compressors are just a bad tool here from a performance standpoint. I’m not aware of a good tool for this kind of application, though.
[1] https://facebook.github.io/zstd/#benchmarks
[2] https://github.com/zlib-ng/zlib-ng/issues/1486
-
CodeRabbit
CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
C zlib-replacement discussion
Index
# | Project | Stars |
---|---|---|
1 | zlib-ng | 1,630 |