LZ4 VS zstd

LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md

According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.

The popular NoSQL database Cassandra utilizes a compression algorithm called LZ4 to reduce the footprint of data at rest. LZ4 is characterized by very fast compression speed at the cost of a higher compression ratio. This is a design choice that allows Cassandra to maintain high write throughput while also benefiting from compression in some capacity.

Yeah, sure, when you have monster core counts. on regular systems, not so much, here's from their own github page. it achieves, eh, 5GB/s on memory to memory transfers, i.e. best case scenario. so, uh, no? i'm not even sure it's any better than the CPU decompressor one Nvidia used.

> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194

There's also LZ4_HC.

Since nobody mentioned it, I'll add lz4 (https://github.com/lz4/lz4).

lz4 can still be the right choice when decompression speed matters. It's almost twice as fast at decompression with similar compression ratios to zstd's fast setting.

https://github.com/facebook/zstd?tab=readme-ov-file#benchmar...

So why would anyone compress an image again by using gzip or zstd?

This is because Zstd's long-distance matcher looks for matching sequences of 64 bytes [0]. Because long matching sequences of the data will likely have the newlines inserted in different offsets in the run, this totally breaks Zstd's ability to find the long-distance match.

Ultimately, Zstd is a byte-oriented compressor that doesn't understand the semantics of the data it compresses. Improvements are certainly possible if you can recognize and separate that framing to recover a contiguous view of the underlying data.

[0] https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zs...

(I am one of the maintainers of Zstd.)

Perhaps it's worth it adding this as a note at the top of the post, maybe mentioning alternatives, such as an Actually Portable™ build of `pigz`[1] or just a windows build of zstd[2].

[1] https://cosmo.zip/pub/cosmos/tiny/pigz

[2] https://github.com/facebook/zstd/releases/latest/

If zstd give you an error and you don't handle it, the next calls may cause UB, so it kinda does both things.

https://github.com/facebook/zstd/blob/b16d193512d3ded82fd584...

( https://github.com/facebook/zstd/releases/tag/v1.5.6 )

In my opinion, it is better to check the original repository: https://github.com/inikep/lzbench

This. Even their less known work is pretty solid[1] ( used it the other day and was frankly kinda amazed at how well it performed under the circumstances ). Facebook/Meta sucks like most social madia does, but, not unlike Elon Musk, they are on the record of having some contributions to society as a whole.

[1]https://github.com/facebook/zstd

I don't think so? It's only seekable with an additional index [1], just like any other compression scheme.

[1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...