Compactor VS zstd

You can already do the very basic version of that by using Compactor. Essentially it uses Windows 10's compression algorithms to make your games smaller. A lot of variance between games based on what their files consist of, but I generally get ~25% more space on my drives. Of course, a single update to the game might mean that you have to pack them again, but it's still a pretty good upside with practically no performance loss.

This will also help.

ARK practically needs its own 500GB SSD, specially if you start installing mods on top of having all of the DLCs. However, it's possible to bring down its size massively if you use this software, it works exceptionally well in ARK, being able to reduce it to about half its size. For example, my ARK installation went from like 320 GB to 175 I believe, which is still huge, but only about as much as any modern AAA game.

You can chuck compact.exe at it by hand (use the /exe option), or if you'd prefer to avoid the command line there's my Compactor tool, or the venerable CompactGUI.

They actually don't compress it. Using a simple compressor like this one for instance, the size decreases by half, if not a bit more than it. Tested it with the base game and it went from 200GB to 100GB.

I should press on and finish my Compactor rewrite, I guess.

7zip, microsoft powertoys (useful utilities n stuff), hyper-v (VM stuff), OBS, optimizer (https://github.com/hellzerg/optimizer) (privacy stuff + system tweaks), compactor (https://github.com/Freaky/Compactor) (uses windows compact.exe and provides a GUI for a otherwise cmd based tool) also qbittorrent

This VFS will read a sqlite file after it has been compressed using [zstd seekable format](https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...). Built to support read-only databases for full-text search. Benchmarks are provided in README.

Of course, you may get different results with another dataset.

gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]

zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]

NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.

NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.

NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases

For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.

xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]

Compression, synchronization and backup systems often use rolling hash to implement "content-defined chunking", an effective form of deduplication.

In optimized implementations, Rabin-Karp is likely to be the bottleneck. See for instance https://github.com/facebook/zstd/pull/2483 which replaces a Rabin-Karp variant by a >2x faster Gear-Hashing.

Get the data https://publicdistst.blob.core.windows.net/data/root.tar.zst magnet:?xt=urn:btih:84931cd80409ba6331f2fcfbe64ba64d4381aec5&dn=root.tar.zst How to extract https://github.com/facebook/zstd Linux (debian): `sudo apt install zstd` ``` tar -I 'zstd -d -T0' -xvf root.tar.zst ```

I've done that experiment with zstd before.

https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md...

Not sure about brotli though.

If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349