zlib-ng
LZ4
zlib-ng | LZ4 | |
---|---|---|
14 | 24 | |
1,630 | 10,592 | |
1.9% | 0.7% | |
9.0 | 9.2 | |
7 days ago | 11 days ago | |
C | C | |
zlib License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zlib-ng
-
Discord Reduced WebSocket Traffic by 40%
For what it’s worth, the benchmark on the Zstandard homepage[1] shows none of the compressors tested breaking 1GB/s on compression, and only the fastest and sloppiest ones breaking 1GB/s on decompression. If you’re OK with its API limitations, libdeflate[2] is known to squeeze past 1GB/s decompressing normal Deflate compression levels. So asking for multiple GB/s is probably unfair.
Still, 10MB/s sounds like the absolute minimum reasonable speed, and they’re reporting nearly three orders of magnitude below that. A modern compressor does not run at bad dialup speeds; something in there is absolutely murdering the performance.
And it might just be the constant-time overhead, as far as I can see. The article mentions “a few hundred bytes” of payload, and the discussion of measurements implies 1.5KB uncompressed. Even though they don’t reinitialize the compressor on each message, that is still a very very modest amount of data.
So it might be that general-purpose compressors are just a bad tool here from a performance standpoint. I’m not aware of a good tool for this kind of application, though.
[1] https://facebook.github.io/zstd/#benchmarks
[2] https://github.com/zlib-ng/zlib-ng/issues/1486
-
Show HN: Pzip- blazing fast concurrent zip archiver and extractor
Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:
https://github.com/zlib-ng/zlib-ng/discussions/871
IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.
- Intel QuickAssist Technology Zstandard Plugin for Zstandard
-
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.
-
Zlib Critical Vulnerability
Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328
Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.
-
Git’s database internals II: commit history queries
I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.
https://github.com/zlib-ng/zlib-ng/discussions/871
-
Computing Adler32 Checksums at 41 GB/s
zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng
Might be interesting to benchmark their implementation too to see how it compares.
-
Convenient CPU feature detection and dispatch in the Magnum Engine
zlib-ng: https://github.com/zlib-ng/zlib-ng/blob/develop/functable.c
-
games-emulation/dolphin-9999 is failing to build because devs switched to minizip-ng and zlib uses minizip. I'm not sure how to get it to build now, details in post.
(2) There are many packages that rely upon zlib and minizip and switching those underlying dependencies is easier said than done. We can't drop zlib completely and switch: "The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change." - https://github.com/zlib-ng/zlib-ng
LZ4
- LZ4 v1.10.0 – Multicores Edition
-
Number sizes for LZ77 compression
LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md
-
Rsyncing 20TB locally
According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.
-
An Intro to Data Compression
The popular NoSQL database Cassandra utilizes a compression algorithm called LZ4 to reduce the footprint of data at rest. LZ4 is characterized by very fast compression speed at the cost of a higher compression ratio. This is a design choice that allows Cassandra to maintain high write throughput while also benefiting from compression in some capacity.
-
Micron Unveils 24GB and 48GB DDR5 Memory Modules | AMD EXPO and Intel XMP 3.0 compatible
Yeah, sure, when you have monster core counts. on regular systems, not so much, here's from their own github page. it achieves, eh, 5GB/s on memory to memory transfers, i.e. best case scenario. so, uh, no? i'm not even sure it's any better than the CPU decompressor one Nvidia used.
- Cerbios Xbox Bios V2.2.0 BETA Released (1.0 - 1.6)
-
zstd
> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.
lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194
There's also LZ4_HC.
-
Best archival/compression format for whole hard drives
Since nobody mentioned it, I'll add lz4 (https://github.com/lz4/lz4).
What are some alternatives?
zstd - Zstandard - Fast real-time compression algorithm
ZLib - A massively spiffy yet delicately unobtrusive compression library.
brotli - Brotli compression format
libdeflate - Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
Snappy - A fast compressor/decompressor
Minizip-ng - Fork of the popular zip manipulation library found in the zlib distribution.
LZMA - (Unofficial) Git mirror of LZMA SDK releases
zlib - Cloudflare fork of zlib with massive performance improvements
LZFSE - LZFSE compression library and command line tool