zstd
brotli
Our great sponsors
zstd | brotli | |
---|---|---|
105 | 25 | |
22,293 | 13,098 | |
1.9% | 1.1% | |
9.6 | 8.3 | |
7 days ago | 9 days ago | |
C | TypeScript | |
GNU General Public License v3.0 or later | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zstd
-
Chrome Feature: ZSTD Content-Encoding
Citation needed? https://github.com/facebook/zstd/commits/dev/?author=jiaT75
Of course, you may get different results with another dataset.
gzip (zlib -6) [ratio=32%] [compr=35Mo/s] [dec=407Mo/s]
zstd (zstd -2) [ratio=32%] [compr=356Mo/s] [dec=1067Mo/s]
NB1: The default for zstd is -3, but the table only had -2. The difference is probably small. The range is 1-22 for zstd and 1-9 for gzip.
NB2: The default program for gzip (at least with Debian) is the executable from zlib. With my workflows, libdeflate-gzip iscompatible and noticably faster.
NB3: This benchmark is 2 years old. The latest releases of zstd are much better, see https://github.com/facebook/zstd/releases
For a high compression, according to this benchmark xz can do slightly better, if you're willing to pay a 10× penalty on decompression.
xz -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
zstd -9 [ratio=23%] [compr=2.6Mo/s] [dec=88Mo/s]
There is an issue tracking this with a bunch of links to discussions about it, but they continue to not have time it seems.
https://github.com/facebook/zstd/issues/3100
This was the first place my mind went when I saw this Content-Encoding announcement, so I ran and re-checked the issue :(.
Yes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors, 300+ other contributors have made 4500+ commits to the zstd repo.
It's not quite as small-scale as 90's-style shareware was.
- Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
-
How in the world should we unpack archive.org zst files on Windows?
If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349
- ZSTD 1.5.5 is released with a corruption fix found at Google
-
Float Compression 3: Filters
Interesting to match with the observations from the practice of using ClickHouse[1][2] for time series:
1. Reordering to SOA helps a lot - this is the whole point of column-oriented databases.
2. Specialized codecs like Gorilla[3], DoubleDelta[4], and FPC[5] lose to simply using ZSTD[6] compression in most cases, both in compression ratio and in performance.
3. Specialized time-series DBMS like InfluxDB or TimescaleDB lose to general-purpose relational OLAP DBMS like ClickHouse [7][8][9].
[1] https://clickhouse.com/blog/optimize-clickhouse-codecs-compr...
[2] https://github.com/ClickHouse/ClickHouse
[3] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[4] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[5] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[6] https://github.com/facebook/zstd/
[7] https://arxiv.org/pdf/2204.09795.pdf "SciTS: A Benchmark for Time-Series Databases in Scientific Experiments and Industrial Internet of Things" (2022)
[8] https://gitlab.com/gitlab-org/incubation-engineering/apm/apm... https://gitlab.com/gitlab-org/incubation-engineering/apm/apm...
[9] https://www.sciencedirect.com/science/article/pii/S187705091...
-
We're wasting money by only supporting gzip for raw DNA files
zstd has a long range mode, which lets it find redundancies a gigabyte away. Try --long and --long=31 for very long range mode.
zstd has delta / patch mode, which creates a file that stores the "patch" to create a new file from an old (reference) file. See https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...
See the man page: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md
brotli
-
Jpegli: A New JPEG Coding Library
JPEGLI = A small JPEG
The suffix -li is used in Swiss German dialects. It forms a diminutive of the root word, by adding -li to the end of the root word to convey the smallness of the object and to convey a sense of intimacy or endearment.
This obviously comes out of Google Zürich.
Other notable Google projects using Swiss German:
https://github.com/google/gipfeli high-speed compression
Gipfeli = Croissant
https://github.com/google/guetzli perceptual JPEG encoder
Guetzli = Cookie
https://github.com/weggli-rs/weggli semantic search tool
Weggli = Bread roll
https://github.com/google/brotli lossless compression
Brötli = Small bread
-
Compression efficiency with shared dictionaries in Chrome
The brotli repo on github has a dictionary generator: https://github.com/google/brotli/blob/master/research/dictio...
I have a hosted version of it on https://use-as-dictionary.com/ to make it easier to experiment with.
-
The Full-Stack development experience
An additional element that we can finally remove from our stack is the minification of JavaScript and CSS files. Thanks to algorithms like brotli (with a very Swiss flavour) we no longer need to minify and compress our files before distributing them. Cloudflare, Nginx, or Apache will take care of everything for us.
-
How Much Faster Is Making a Tar Archive Without Gzip?
For anyone who wants to try this, zstd -T0 uses all your threads to compress, and https://github.com/facebook/zstd has a lot more description. Brotli, https://github.com/google/brotli, is another modern format with some good features for high compression levels and Content-Encoding support in web browsers. You might also want to play with the compression level (-1 to -11 or more, zstd's --fast=n).
One reason these modern compressors do better is not any particular mistake made defining DEFLATE in the 90s, but that new algos use a few MB of recently seen data as context instead of 32KB, and do other things impractical in the 90s but reasonable on modern hardware. The new algorithms also contain logs of smart ideas and have fine-tuned implementations, but that core difference seems important to note.
-
Troubling Launching Duckstation
It seems to be using a lib called brotli - https://github.com/google/brotli. Can you compile from source?
-
2000’s Winamp/WMP-ish skins?
(++) Play Youtube videos and playlistsRelease date: Feb 3 20223dyd, [email protected] libraries: brotli https://github.com/google/brotli (1.0.7)
-
Stop delaying. Share knowledge on a blog built with Eleventy.
Alright, the minifying is done. What else? Did you know you can serve HTML, CSS and JS compressed? A lot of websites still use gzip, but there’s also Brotli. Brotli is specifically made for the web and compresses a lot better than gzip in most cases.
-
Configuring CloudFront to compress objects in AWS CDK
You can use CloudFront to automatically compress certain types of objects (files) and serve the compressed objects when viewers (web browsers or other clients) support them. Viewers indicate their support for compressed objects with the Accept-Encoding HTTP header. CloudFront can compress objects using the Gzip and Brotli compression formats. When the viewer supports both formats, CloudFront prefers Brotli.
-
Looking for maintainer for jvm-brotli
Hi /r/java! Jetty is considering implementing dynamic Brotli compression, but the current JVM wrapper for Google's Brotli (jvm-brotli) is somewhat ... abandoned.
What are some alternatives?
LZ4 - Extremely Fast Compression algorithm
Snappy - A fast compressor/decompressor
LZMA - (Unofficial) Git mirror of LZMA SDK releases
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
ZLib - A massively spiffy yet delicately unobtrusive compression library.
haproxy - HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
LZFSE - LZFSE compression library and command line tool
zlib-ng - zlib replacement with optimizations for "next generation" systems.