zstd
zlib-ng
Our great sponsors
zstd | zlib-ng | |
---|---|---|
105 | 13 | |
22,356 | 1,440 | |
2.2% | 1.8% | |
9.7 | 9.4 | |
8 days ago | 14 days ago | |
C | C | |
GNU General Public License v3.0 or later | zlib License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zstd
-
Chrome Feature: ZSTD Content-Encoding
Of course, you may get different results with another dataset.
gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.
NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.
NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases
For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.
xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
- Zstandard v1.5.6 – Chrome Edition
-
Optimizating Rabin-Karp Hashing
Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.
In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.
- Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
-
Cyberpunk 2077 dev release
Get the data https://publicdistst.blob.core.windows.net/data/root.tar.zst magnet:?xt=urn:btih:84931cd80409ba6331f2fcfbe64ba64d4381aec5&dn=root.tar.zst How to extract https://github.com/facebook/zstd Linux (debian): `sudo apt install zstd` ``` tar -I 'zstd -d -T0' -xvf root.tar.zst ```
-
Honey, I shrunk the NPM package · Jamie Magee
I've done that experiment with zstd before.
https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...
Not sure about brotli though.
-
How in the world should we unpack archive.org zst files on Windows?
If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349
- Release Zstandard v1.5.5 · facebook/zstd
- ZSTD 1.5.5 is released with a corruption fix found at Google
-
zstd is used at Google
The story says : "ZSTD 1.5.5 is released with a corruption fix found at Google"
zlib-ng
-
Show HN: Pzip- blazing fast concurrent zip archiver and extractor
Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:
https://github.com/zlib-ng/zlib-ng/discussions/871
IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.
- Intel QuickAssist Technology Zstandard Plugin for Zstandard
-
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.
-
Zlib Critical Vulnerability
Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328
Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.
-
Git’s database internals II: commit history queries
I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.
https://github.com/zlib-ng/zlib-ng/discussions/871
-
Computing Adler32 Checksums at 41 GB/s
zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng
Might be interesting to benchmark their implementation too to see how it compares.
-
Convenient CPU feature detection and dispatch in the Magnum Engine
zlib-ng: https://github.com/zlib-ng/zlib-ng/blob/develop/functable.c
-
games-emulation/dolphin-9999 is failing to build because devs switched to minizip-ng and zlib uses minizip. I'm not sure how to get it to build now, details in post.
(2) There are many packages that rely upon zlib and minizip and switching those underlying dependencies is easier said than done. We can't drop zlib completely and switch: "The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change." - https://github.com/zlib-ng/zlib-ng
-
Re: Zlib memory corruption on deflate (i.e. compress)
There are already active zlib forks (e.g. https://github.com/zlib-ng/zlib-ng), the problem is with having people move to them. It takes a lot of effort to move mindshare from the original version to a fork, there's some historical examples of it happening, but not a ton.
What are some alternatives?
LZ4 - Extremely Fast Compression algorithm
ZLib - A massively spiffy yet delicately unobtrusive compression library.
Snappy - A fast compressor/decompressor
Minizip-ng - Fork of the popular zip manipulation library found in the zlib distribution.
LZMA - (Unofficial) Git mirror of LZMA SDK releases
libdeflate - Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
brotli - Brotli compression format
uzlib - Radically unbloated DEFLATE/zlib/gzip compression/decompression library. Can decompress any gzip/zlib data, and offers simplified compressor which produces gzip-compatible output, while requiring much less resources (and providing less compression ratio of course).