LZ4
zstd
Our great sponsors
LZ4 | zstd | |
---|---|---|
10 | 54 | |
7,123 | 16,909 | |
3.3% | 1.5% | |
7.8 | 9.6 | |
4 days ago | 1 day ago | |
C | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
LZ4
-
Bzip3 – a better and stronger spiritual successor to bzip2
If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy.
-
Activision says Warzone doesn’t support map rotation because of ‘crazy install sizes’ | VGC
gonna go off-topic, but doesn't LZ4 exists? I know IOI Interactive uses it for HITMAN 3.
-
I just took a random screenshot and made it look prettier. [ I don't know if this counts as fanart ]
E: Realtime compression (A good compression library like Zstandard can make a game less than half the size while taking a tiny amount of CPU power when loading stuff. I think thats a pretty worthwhile trade.) (ZSTD github) (LZ4 github)
-
What's the best way to compress strings?
lz4 for maximum decompression speed, for data that is often read but rarely written
-
Exomizer & Graforth
That's great news. I implemented LZ4 for the Apple II, but Yann Collet is the creator of the algorithm.
-
How to become a tools/graphics/engine programmer
Getting lost in material models is tempting. But, at this point you are overdue for working on your own asset pipeline. glTF is great. But, you should learn how to do it yourself. The hardest part will be reading source asset files. The FBX SDK is painful. Assimp isn't great either. Writing your own exporter to your own intermediate text format from Maya or Blender would be good if you are up for it. From whatever source, make your own archive format and binary formats for meshes, animations, textures and scenes. Use https://github.com/lz4/lz4 for compression. You should be able to decompress a list of assets into a big linear array and use them right there with just a bit of pointer fix-up. Minimize the amount of memory you have to touch from start to finish. Data that is going to the GPU (textures, vertex/index buffers) should decompress straight into mapped buffers for fast uploads.
-
Activision warns a standard 500GB PS4 may no longer fit Call of Duty: Warzone, Black Ops Cold War and Modern Warfare
Oodle Kraken and lz4 are a thing.
-
LZ4, an Extremely Fast Compression Algorithm
I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4
It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio
-
lz4_flex 0.7.2 reaches parity with cpp reference implementation on speed and ratio
It's hard to tell, how much since the latest lz4 1.9 has been improved, because the (benchmarks)[https://github.com/lz4/lz4/commit/31763c59fca9a38bc697cd37f591d6ba385f7614#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5] are executed with different processors
Which version are you using ? Is that the reference implementation at github.com/lz4/lz4 ?
zstd
-
Lizard – efficient compression with fast decompression
The thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd
But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
-
Zstandard Worked Example
https://github.com/facebook/zstd/tree/dev/doc/educational_de... is a self-contained zstd decoder. I get a 64 KB dynamically linked executable after running "make" in that directory.
$ size harness
> Yikes; half a meg of code!
It's plausible that the lib you checked is the output from the project's default build target (zstd), which "(...) uncludes dictionary builder, benchmark, and supports decompression of legacy zstd formats"
https://github.com/facebook/zstd/tree/dev/programs
The project also provides another build target, zstd-small, which is "CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats"
Also, take a look at what exactly is bundled with the binary.
Thanks for the feedback! I've opened an issue to track this [0]
* Levels 1-19 are the "standard" compression levels.
* Levels 20-22 are the "ultra" levels which require --ultra to use on the CLI. They allocate a lot of memory and are very slow.
* Level 0 is the default compression level, which is 3.
* Levels < 0 are the "fast" compression levels. They achieve speed by turning off Huffman compression, and by "accelerating" compression by a factor. Level -1 has acceleration factor 1, -2 has acceleration factor 2, and so on. So the minimum supported negative compression level is -131072, since the maximum acceleration factor is our block size. But in practice, I wouldn't think a negative level lower than -10 or -20 would be all that useful.
The first 4 bytes are the magic number and the last 4 bytes are the checksum [1] which you could always just chop off if you wanted (it's legal to omit the checksum, see the spec). That would get the total overhead down to 5 bytes.
[1]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compressi...
The official cli windows binaries are here: https://github.com/facebook/zstd/releases/
Afaik, they don't need any dependencies.
-
Choose wisely
gzip certainly doesn't help. Switching to zstd should be an easy bit of low-hanging fruit, at least for save times.
-
Casync – A Content-Addressable Data Synchronization Tool
Really wish this was part of official zstd (https://github.com/facebook/zstd/issues/395#issuecomment-535...) and not a contrib / separate tool.
What are some alternatives?
Snappy - A fast compressor/decompressor
LZMA - (Unofficial) Git mirror of LZMA SDK releases
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
ZLib - A massively spiffy yet delicately unobtrusive compression library.
brotli - Brotli compression format
zfs - OpenZFS on Linux and FreeBSD
LZFSE - LZFSE compression library and command line tool
LZHAM - Lossless data compression codec with LZMA-like ratios but 1.5x-8x faster decompression speed, C/C++