libdeflate
TurboBench
Our great sponsors
libdeflate | TurboBench | |
---|---|---|
7 | 10 | |
931 | 310 | |
- | - | |
8.8 | 8.9 | |
15 days ago | 9 months ago | |
C | C | |
MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
libdeflate
- Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
-
Introducing zune-inflate: The fastest Rust implementation of gzip/Zlib/DEFLATE
zune-inflate is a port of libdeflate to safe Rust.
-
Quite OK Image is now my favorite asset format
> The QOI benchmarks shows it only lagging a small bit behind PNG in compression ratio, while performing much faster (multiples) in both encoding and decoding.
Presumably that’s with stock libpng, which uses zlib. I wonder if anyone tried patching it to use the substantially faster (on x86) libdeflate[1] instead? It doesn’t do streaming, but you shouldn’t really need it for PNG textures.
[1] https://github.com/ebiggers/libdeflate
-
bitter v0.6 - the performant bit reading library with a new faster API
Eric biggers libdeflate https://github.com/ebiggers/libdeflate/blob/master/lib/deflate_decompress.c
-
Computing Adler32 Checksums at 41 GB/s
Note that libdeflate has used essentially the same method since 2016 (https://github.com/ebiggers/libdeflate/blob/v0.4/lib/adler32...), though I recently switched it to use a slightly different method (https://github.com/ebiggers/libdeflate/blob/v1.12/lib/x86/ad...) that performs more consistently across different families of x86 CPUs.
-
Zlib – a spiffy yet delicately unobtrusive compression library
Going away from Zlib proper but staying with its format, there’s libdeflate[1], an implementation that’s substantially faster on modern machines but can’t stream, and of course Zopfli[2], a glacially slow compressor that’s somewhat better (and occasionally much more so).
[1] https://github.com/ebiggers/libdeflate
[2] https://github.com/google/zopfli
- Firefox's Optimized Zip Format: Reading Zip Files Quickly
TurboBench
-
Zstd Content-Encoding planned to ship with Chrome 123
I'm still unconvinced about this addition. And I don't even dislike Zstandard.
The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2].
According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead. In fact, Meta employs the main author of Zstandard so it would have been easier to tune Zstandard instead of Brotli. While Brotli has some fundamental difference from Zstandard (in particular Brotli doesn't use arithmetic-equivalent coding), no one has concretely demonstrated that difference would prevent Brotli from being fast enough for dynamic contents in my opinion.
[1] https://issues.chromium.org/issues/40196713
[2] https://github.com/powturbo/TurboBench/issues/43
[3] https://docs.google.com/document/d/14dbzMpsYPfkefAJos124uPrl...
- TurboBench: Dynamic/Static web content compression benchmark
-
Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
libdeflate compress better and has faster decompression than igzip.
See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,...
https://github.com/powturbo/TurboBench/issues/4
-
Intel QuickAssist Technology Zstandard Plugin for Zstandard
- https://github.com/powturbo/TurboBench/issues/43
[1] https://github.com/powturbo/TurboBench
-
Variation on RLE to Achieve Lossless Compression for Tabular Data
Compressesing your sample file, we get 823 bytes with brotli
Download TurboBench and make your own tests:
[1] - https://github.com/powturbo/TurboBench
-
Data Compression Drives the Internet. Here’s How It Works
- igzip 1,2 is best for very fast networks > 10MB/s
brotli bring little value at decompression for users
[1] https://github.com/powturbo/TurboBench
[1] https://sites.google.com/site/powturbo/home/web-compression
[2] https://encode.su/threads/2333-TurboBench-Back-to-the-future...
-
Pigz: Parallel gzip for modern multi-processor, multi-core machines
Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors.
[1] https://github.com/powturbo/TurboBench
[2] https://github.com/powturbo/TurboBench/releases
What are some alternatives?
zlib-ng - zlib replacement with optimizations for "next generation" systems.
QAT-ZSTD-Plugin
ZLib - A massively spiffy yet delicately unobtrusive compression library.
rapidgzip - Gzip Decompression and Random Access for Modern Multi-Core Machines
zlib - Cloudflare fork of zlib with massive performance improvements
pigz - A parallel implementation of gzip for modern multi-processor, multi-core machines.
zlib
lib842
fpng - Super fast C++ .PNG writer/reader
DirectStorage - DirectStorage for Windows is an API that allows game developers to unlock the full potential of high speed NVMe drives for loading game assets.
HexFiend - A fast and clever hex editor for macOS
FPC - FPC - Fast Prefix Coder