LZ4 VS ZLib

According to these https://github.com/lz4/lz4 values you need around ten (10) quite modern cores in parallel to accomplish around 8GB/s.

> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194

There's also LZ4_HC.

Get your bootloader unlocked via Download mode and then obtain your stock firmware, preferably for your current region https://samfw.com (Download mode: CARRIER_CODE). Get the boot image from AP with 7zip, unpack from LZ4 with https://github.com/lz4/lz4/releases (drag and drop), patch with Magisk https://github.com/topjohnwu/magisk/releases/latest, grab the new image, name it "boot.img" and pack it into a .tar with 7zip and flash to AP with odin https://odindownload.com

After some investigations and experiments, I found out that it was the PNG compression (well, decompression I should say) that took a while. So I've made some experiments using the LZ4 compression library, which is focused on decompression speed, and it turned out to be an excellent solution!

If anyone just cares for speed instead of compression I’d recommend lz4 [1]. I only recently started using it. Its speed is almost comparable to memcpy.

[1] https://github.com/lz4/lz4

E: Realtime compression (A good compression library like Zstandard can make a game less than half the size while taking a tiny amount of CPU power when loading stuff. I think thats a pretty worthwhile trade.) (ZSTD github) (LZ4 github)

lz4 for maximum decompression speed, for data that is often read but rarely written

Getting lost in material models is tempting. But, at this point you are overdue for working on your own asset pipeline. glTF is great. But, you should learn how to do it yourself. The hardest part will be reading source asset files. The FBX SDK is painful. Assimp isn't great either. Writing your own exporter to your own intermediate text format from Maya or Blender would be good if you are up for it. From whatever source, make your own archive format and binary formats for meshes, animations, textures and scenes. Use https://github.com/lz4/lz4 for compression. You should be able to decompress a list of assets into a big linear array and use them right there with just a bit of pointer fix-up. Minimize the amount of memory you have to touch from start to finish. Data that is going to the GPU (textures, vertex/index buffers) should decompress straight into mapped buffers for fast uploads.

I'm not a fan of the stacked bar charts, I like the table of data for "Benchmarks" on the github source page: https://github.com/lz4/lz4

It makes it very clear where LZ4 fits into comparisons with compression speed, decompression speed and compression ratio

A minimal viable Deflate decompressor is not exactly complex[1], although slower than mainline zlib.

[1] https://github.com/madler/zlib/tree/master/contrib/puff

ZLibrary not zlib for anyone worried about archiving algorithms. https://zlib.net/ and all the Linux repositories are safe (AFAIK).

>The following is my educated guess at what to do next and I am a week-end warrior programmer at best. If someone with more experience comments, I would listen to them over me< If you have no programming experience, this may be a lost cause. It looks like the save file has a header area followed by a list of "zlib" compressed chunks. zlib.net may have a program that can open the save file and let you see the raw information inside it. Something after the header is garbled and it is preventing Save-Editors from decompressing that chunk OR from parsing what is inside of it. If you are lucky, the garbled bit will jump out at you and you can repair it.

These appears to be the relevant changes:

2022-07-30: https://github.com/madler/zlib/commit/eff308af425b67093bab25...

2022-08-08: https://github.com/madler/zlib/commit/1eb7682f845ac9e9bf9ae3...

The second commit definitely fixed a null pointer dereference, I am not sure if the CVE is referencing something else that was fixed by the first commit.

It is actually possible to estimate the compression level without actually compressing everything again, because different levels use different strategies which can be identified in some cases. zlib in particular has three strategies [1] and preflate [2] leverages this to store the deflate stream reconstruction data without much bits.

[1] https://github.com/madler/zlib/blob/21767c6/deflate.c#L134-L...

[2] https://github.com/deus-libri/preflate/blob/master/preflate_...

It's still a heap of old school C spaghetti https://github.com/madler/zlib/commit/eff308af425b67093bab25...

The non-forked zlib hasn't been accepting optimisations: https://github.com/madler/zlib/issues/346

Hopefully changes will be merged from my obscure private fork into four other obscure private forks (zlib-chromium (https://crbug.com/1354990), zlib-ng, zlib-cloudflare, and the one I personally care about, which doesn't take pull requests, zlib-apple), and possibly incorporated into libdeflate. That's the point of the blog post - to help maintainers understand the changes.

This seems to be a minor bug, in that it is only triggered using unusual (and rather unlikely) deflate parameters.

But, looking into this bug, I was sort-of interested to see how it was handled. The change log for 1.2.12 (2022-03-27) indicates the issue was resolved ("Fix a bug that can crash deflate on some input when using Z_FIXED"). Yet, in what seems to be the canonical Zlib repository (https://github.com/madler/zlib), I was unable to find a corresponding commit.

None of commits this year (7, so not too hard to review) seem to be particularity meaningful changes, and in particular yesterday's 'zlib 1.2.12' commit seems to only consist of version/copyright updates.

So, does anyone have any idea where to find the commits related to the change log entry? (Note that I'm not disputing the issue is actually fixed, I'm just trying to improve my Github reading skills...)