Snappy VS brotli

Another example of Nim being really fast is the supersnappy library. This library benchmarks faster than Google’s C or C++ Snappy implementation.

It doesn't destroy performance for the simple reason that nowadays memory access has higher latency than pure compute. If you need to use compute to produce some data to be stored in memory, your overall throughput could very well be faster than without compression.

There have been a large amount of innovation on fast compression in recent years. Traditional compression tools like gzip or xz are geared towards higher compression ratio, but memory compression tends to favor speed. Check out those algorithms:

* lz4: https://lz4.github.io/lz4/

* Google's snappy: https://github.com/google/snappy

* Facebook's zstd in fast mode: http://facebook.github.io/zstd/#benchmarks

Google released Snappy, which is extremely fast and robust (both at compression and decompression), but it's definitely not nearly as good (in terms of compression ratio). Google mostly uses it for real-time compression, for example of network messages - not for long-term storage.

Just compress it! Of course if you will you ZIP, players will able to just open this zip file and change whatever they want. But you can use less popular compression algorithms which are not supported by default Windows File Explorer. Snappy for example.

Projects like Brotli aren't built to maximize personal profit; they're driven by passion and a genuine love for software engineering.

It's clear that the industry is shifting from being geeky and nerdy to being more business and management focused.

[0] https://github.com/google/brotli

Brotli is a compression algorithm developed by Google, particularly effective for text and web compression. It uses a combination of LZ77 (Lempel-Ziv 77), Huffman coding, and 2nd order context modeling. In comparison to traditional algorithms like Gzip, Brotli can achieve smaller compressed sizes, especially for HTML and text-heavy content. This makes it a good candidate for our Pied Piper-inspired text compression implementation.

The one example I can think of with a pre-seeded dictionary (for web, no less) is Brotli.

https://datatracker.ietf.org/doc/html/rfc7932#appendix-A

You can more or less see what it looks like (per an older commit): https://github.com/google/brotli/blob/5692e422da6af1e991f918...

Certainly it performs better than gzip by itself.

Some historical discussion: https://news.ycombinator.com/item?id=19678985

I believe the compression dictionary refers to [1], which is used to quickly match dictionary-compressable byte sequences. I don't know where 170 KB comes from, but that hash alone does take 128 KiB and might be significant if it can't be easily recomputed. But I'm sure that it can be quickly computed on the loading time if the binary size is that important.

[1] https://github.com/google/brotli/blob/master/c/enc/dictionar...

The answer lies in the Accept-Encoding header. In the example above, I just copied it from my browser, so it lists all the compression methods my browser supports: "gzip, deflate, br, zstd". The Wayfair backend supports compression with "br", which is Brotli, and uses it as the most efficient method.

...though with the slightly unexpected side effect (for Brotli, at least) that your executable may end up containing (~200KB, from memory) of very unexpected plain text strings which might (& has[0]) lead to questions from software end-users asking why your software contains "random"[1] text (including potentially "culturally sensitive" words/phrases related to religion such as "Holy Roman Emperor", "Muslims", "dollars", "emacs"[2] or similar).

(I encountered this aspect while investigating potential size optimization opportunities for the Godot game engine's web/WASM builds--though presumably the Brotli dictionary compresses well if the transfer encoding is... Brotli. :D )

[0] "This needs to be reviewed immediately #876": https://github.com/google/brotli/issues/876

[1] Which, regardless of meaning, certainly bears similarities to the type of "unexpected weird text" commonly/normally associated with spam, malware, LLMs and other entities of ill repute.

[2] The final example may not actually be factual. :)

Using tools like Brotli, you can boost your application’s load time. You can use the ngUpgrade library to mix AngularJS and Angular components to enhance runtime performance, bringing in hybrid applications that can be used with techniques like ahead-of-time (AOT) compilation, aiding in faster browser rendering.

JPEGLI = A small JPEG

The suffix -li is used in Swiss German dialects. It forms a diminutive of the root word, by adding -li to the end of the root word to convey the smallness of the object and to convey a sense of intimacy or endearment.

This obviously comes out of Google Zürich.

Other notable Google projects using Swiss German:

https://github.com/google/gipfeli high-speed compression

Gipfeli = Croissant

https://github.com/google/guetzli perceptual JPEG encoder

Guetzli = Cookie

https://github.com/weggli-rs/weggli semantic search tool

Weggli = Bread roll

https://github.com/google/brotli lossless compression

Brötli = Small bread

The brotli repo on github has a dictionary generator: https://github.com/google/brotli/blob/master/research/dictio...

I have a hosted version of it on https://use-as-dictionary.com/ to make it easier to experiment with.

An additional element that we can finally remove from our stack is the minification of JavaScript and CSS files. Thanks to algorithms like brotli (with a very Swiss flavour) we no longer need to minify and compress our files before distributing them. Cloudflare, Nginx, or Apache will take care of everything for us.