LZ4 VS ZLib

LZ4 is a bit more complicated, but seems faster: https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md

According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.

The popular NoSQL database Cassandra utilizes a compression algorithm called LZ4 to reduce the footprint of data at rest. LZ4 is characterized by very fast compression speed at the cost of a higher compression ratio. This is a design choice that allows Cassandra to maintain high write throughput while also benefiting from compression in some capacity.

Yeah, sure, when you have monster core counts. on regular systems, not so much, here's from their own github page. it achieves, eh, 5GB/s on memory to memory transfers, i.e. best case scenario. so, uh, no? i'm not even sure it's any better than the CPU decompressor one Nvidia used.

> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194

There's also LZ4_HC.

Since nobody mentioned it, I'll add lz4 (https://github.com/lz4/lz4).

zlib

A fix has been checked into the upstream git repo: https://github.com/madler/zlib/pull/843 but a release has not yet been made including it.

So the real issue here is that the lack of tree validation before the tree construction, I believe. I'm surprised that this check was not yet implemented (I actually checked libwebp to make sure that I was missing one). Given this blind spot, an automated test based on the domain knowledge is likely useless to catch this bug.

[1] https://github.com/madler/zlib/blob/master/examples/enough.c

In the source code of the Node.js opensource project, lib folder contains JavaScript code, mostly wrappers over C++ and function definitions. On the contrary, src folder contains C++ implementations of the functions, which pulls dependencies from the V8 project, the libuv project, the zlib project, the llhttp project, and many more - which are all placed at the deps folder.