zlib-ng
fpng
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zlib-ng
-
Show HN: Pzip- blazing fast concurrent zip archiver and extractor
Please note that allowing for 2% bigger resulting file could mean huge speedup in these circumstances even with the same compression routines, seeing these benchmarks of zlib and zlib-ng for different compression levels:
https://github.com/zlib-ng/zlib-ng/discussions/871
IMO the fair comparison of the real speed improvement brought by a new program is only between the almost identical resulting compressed sizes.
- Intel QuickAssist Technology Zstandard Plugin for Zstandard
-
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
It is much faster than miniz_oxide and all other safe-Rust implementations, and consistently beats even Zlib. The performance is roughly on par with zlib-ng - sometimes faster, sometimes slower. It is not (yet) as fast as the original libdeflate in C.
-
Zlib Critical Vulnerability
Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328
Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.
-
Git’s database internals II: commit history queries
I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.
-
Computing Adler32 Checksums at 41 GB/s
zlib-ng also has adler32 implementations optimized for various architectures: https://github.com/zlib-ng/zlib-ng
Might be interesting to benchmark their implementation too to see how it compares.
-
Convenient CPU feature detection and dispatch in the Magnum Engine
zlib-ng: https://github.com/zlib-ng/zlib-ng/blob/develop/functable.c
-
games-emulation/dolphin-9999 is failing to build because devs switched to minizip-ng and zlib uses minizip. I'm not sure how to get it to build now, details in post.
(2) There are many packages that rely upon zlib and minizip and switching those underlying dependencies is easier said than done. We can't drop zlib completely and switch: "The idea of zlib-ng is not to replace zlib, but to co-exist as a drop-in replacement with a lower threshold for code change." - https://github.com/zlib-ng/zlib-ng
-
Re: Zlib memory corruption on deflate (i.e. compress)
There are already active zlib forks (e.g. https://github.com/zlib-ng/zlib-ng), the problem is with having people move to them. It takes a lot of effort to move mindshare from the original version to a fork, there's some historical examples of it happening, but not a ton.
fpng
-
png crate gets an ultrafast compression mode, up to 4x faster decompression
When the QOI format was first announced it wasn't clear that was even possible while keeping PNG format compatibility. But the fpng and fpnge C/C++ libraries showed it was, and today you can take advantage of those advances in a general purpose PNG library in Rust!
-
Hello, PNG
CRC is a table and 5 lines of code. That's trivial.
>zlib is 23k lines
It's not needed to make a PNG reader/writer. zlib is massive overkill for only making a PNG reader or writer. Here's a tiny deflate/inflate code [2] under 1k lines (and could be much smaller if needed).
stb[0] has single headers of ~7k lines total including all of the formats PNG, JPG, BMP,. PSD, GIF, HDR, and PIC. Here's [1] a 3k lines single file PNG version with tons if #ifdefs for all sorts of platforms. Removing those and I'd not be surprised if you could not do it in ~1k lines (which I'd consider quite simple compared to most of todays' media formats).
>Of course they're not common formats so you're stuck with complex formats like PNG
BMP is super common and easy to use anywhere.
I use flat image files all the time for quick and dirty stuff. They quickly saturate disk speeds and networking speeds (say recording a few decent speed cameras), and I've found PNG compression to alleviate those saturate CPU speeds (some libs are super slow, some are vastly faster). I've many times made custom compression formats to balance these for high performance tools when neither things like BMPs or things like PNG would suffice.
[0] https://github.com/nothings/stb
[1] https://github.com/richgel999/fpng/blob/main/src/fpng.cpp
The space cost is negligible, but the time cost for the encoder is real. Take fpng[1] as an example, which tries to push the boundaries of PNG encode speed.
> The above benchmarks were made before SSE adler32/crc32 functions were added to the encoder. With 24bpp images and MSVC2022 the encoder is now around 15% faster.
I can't see the total percentage cost of checksums mentioned anywhere on the page, but we can infer that it's at least 15% of the overall CPU time, on platforms without accelerated checksum implementations.
- Quite OK Image is now my favorite asset format
-
Computing Adler32 Checksums at 41 GB/s
This was actually considered, and other libraries do ignore checksums, or at least have options to:
-
QOI – The Quite OK Image Format
In the other direction, you can target a subset of PNG to get less optimized images but with QOI-like encode and decode speed: https://github.com/richgel999/fpng
-
QOI – The “Quite OK Image Format” for fast, lossless image compression
If QOI is interesting because of speed, you might take a look at fpng, a recent/actively developed png reader/writer that is achieving comparable speed/compression to QOI, while staying png compliant.
https://github.com/richgel999/fpng
Disclaimer: have not actively tried either.
I think QOI inspired the creation of https://github.com/richgel999/fpng which creates standard PNGs and compares itself directly to QOI.
-
Losslessly compresses RGB and RGBA images to a similar size of PNG, while offering a 20x-50x speedup in compression and 3x-4x speedup in decompression
BTW, today I found this fpng-fast PNG writer. There is a comparison with QOI in the readme.
What are some alternatives?
zstd - Zstandard - Fast real-time compression algorithm
ZLib - A massively spiffy yet delicately unobtrusive compression library.
Minizip-ng - Fork of the popular zip manipulation library found in the zlib distribution.
libdeflate - Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
brotli - Brotli compression format
uzlib - Radically unbloated DEFLATE/zlib/gzip compression/decompression library. Can decompress any gzip/zlib data, and offers simplified compressor which produces gzip-compatible output, while requiring much less resources (and providing less compression ratio of course).
LZ4 - Extremely Fast Compression algorithm
Snappy - A fast compressor/decompressor
qoi - The “Quite OK Image Format” for fast, lossless image compression
tiny_jpeg.h - Single header lib for JPEG encoding. Public domain. C99. stb style.
FiniteStateEntropy - New generation entropy codecs : Finite State Entropy and Huff0
LZHAM - Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++