isa-l VS TurboBench

For using accelerators (QAT, I assume, because the article is about this) from the CLI, then QATzip[3] comes with a command line tool qatzip which can be used as described in the project ReadMe. I didn't test it, though, as I have no QAT-enabled device.

[1] https://github.com/intel/isa-l/tree/master/igzip

Ex 1: https://github.com/intel/isa-l/blob/master/README.md

Examples: - Someone I know created machine-optimized library in c and asm to accelerate compression, erasure coding, and encryption on Intel Xeon CPUs (https://github.com/intel/isa-l) which runs those ops WAY faster than the standard linux libraries. - My friend is making a retro SEGA genesis game using a C and asm dev kit (https://github.com/Stephane-D/SGDK)

igzip (https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files.

However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.

I'm still unconvinced about this addition. And I don't even dislike Zstandard.

The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2].

According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead. In fact, Meta employs the main author of Zstandard so it would have been easier to tune Zstandard instead of Brotli. While Brotli has some fundamental difference from Zstandard (in particular Brotli doesn't use arithmetic-equivalent coding), no one has concretely demonstrated that difference would prevent Brotli from being fast enough for dynamic contents in my opinion.

[1] https://issues.chromium.org/issues/40196713

[2] https://github.com/powturbo/TurboBench/issues/43

[3] https://docs.google.com/document/d/14dbzMpsYPfkefAJos124uPrl...

libdeflate compress better and has faster decompression than igzip.

See the silesia single core in-memory benchmark here [1] comparing zlib,libdeflate,igzip,...

https://github.com/powturbo/TurboBench/issues/4

- https://github.com/powturbo/TurboBench/issues/43

[1] https://github.com/powturbo/TurboBench

Compressesing your sample file, we get 823 bytes with brotli

Download TurboBench and make your own tests:

[1] - https://github.com/powturbo/TurboBench

- igzip 1,2 is best for very fast networks > 10MB/s

brotli bring little value at decompression for users

[1] https://github.com/powturbo/TurboBench

[1] https://sites.google.com/site/powturbo/home/web-compression

[2] https://encode.su/threads/2333-TurboBench-Back-to-the-future...

Build or download TurboBench [1] executables for linux and windows from releases [2] ans make your own tests comparing oodle,zstd and other compressors.

[1] https://github.com/powturbo/TurboBench

[2] https://github.com/powturbo/TurboBench/releases