zstd
ZLib
Our great sponsors
zstd | ZLib | |
---|---|---|
101 | 49 | |
22,045 | 5,219 | |
2.0% | - | |
9.6 | 8.9 | |
7 days ago | 3 days ago | |
C | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
zstd
- Show HN: macOS-cross-compiler – Compile binaries for macOS on Linux
-
How in the world should we unpack archive.org zst files on Windows?
If you want this functionality in zstd itself, check this out: https://github.com/facebook/zstd/pull/2349
- ZSTD 1.5.5 is released with a corruption fix found at Google
-
Float Compression 3: Filters
Interesting to match with the observations from the practice of using ClickHouse[1][2] for time series:
1. Reordering to SOA helps a lot - this is the whole point of column-oriented databases.
2. Specialized codecs like Gorilla[3], DoubleDelta[4], and FPC[5] lose to simply using ZSTD[6] compression in most cases, both in compression ratio and in performance.
3. Specialized time-series DBMS like InfluxDB or TimescaleDB lose to general-purpose relational OLAP DBMS like ClickHouse [7][8][9].
[1] https://clickhouse.com/blog/optimize-clickhouse-codecs-compr...
[2] https://github.com/ClickHouse/ClickHouse
[3] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[4] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[5] https://clickhouse.com/docs/en/sql-reference/statements/crea...
[6] https://github.com/facebook/zstd/
[7] https://arxiv.org/pdf/2204.09795.pdf "SciTS: A Benchmark for Time-Series Databases in Scientific Experiments and Industrial Internet of Things" (2022)
[8] https://gitlab.com/gitlab-org/incubation-engineering/apm/apm... https://gitlab.com/gitlab-org/incubation-engineering/apm/apm...
[9] https://www.sciencedirect.com/science/article/pii/S187705091...
-
We're wasting money by only supporting gzip for raw DNA files
zstd has a long range mode, which lets it find redundancies a gigabyte away. Try --long and --long=31 for very long range mode.
zstd has delta / patch mode, which creates a file that stores the "patch" to create a new file from an old (reference) file. See https://github.com/facebook/zstd/wiki/Zstandard-as-a-patchin...
See the man page: https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md
-
Decompressing the ZST files on Windows tips
So I downloaded the Facebook tool https://github.com/facebook/zstd
-
zstd
They have a nice table on that page: https://github.com/facebook/zstd#benchmarks
Looking at that table, I think LZ4 is a winner. The compression ratio is not too far, compression speed is slightly faster, decompression speed is significantly faster, the code is much simpler so the compiled binary is smaller, and the project is unrelated to Facebook.
ZLib
-
ZLib VS jdeflate - a user suggested alternative
2 projects | 25 Nov 2023
-
CVE-2023-4863: Heap buffer overflow in WebP (Chrome)
So the real issue here is that the lack of tree validation before the tree construction, I believe. I'm surprised that this check was not yet implemented (I actually checked libwebp to make sure that I was missing one). Given this blind spot, an automated test based on the domain knowledge is likely useless to catch this bug.
[1] https://github.com/madler/zlib/blob/master/examples/enough.c
-
Notes: Advanced Node.js Concepts by Stephen Grider
In the source code of the Node.js opensource project, lib folder contains JavaScript code, mostly wrappers over C++ and function definitions. On the contrary, src folder contains C++ implementations of the functions, which pulls dependencies from the V8 project, the libuv project, the zlib project, the llhttp project, and many more - which are all placed at the deps folder.
- Exploring the Internals of Linux v0.01
-
Quite OK Image is now my favorite asset format
A minimal viable Deflate decompressor is not exactly complex[1], although slower than mainline zlib.
-
Zlib alternatives?
ZLibrary not zlib for anyone worried about archiving algorithms. https://zlib.net/ and all the Linux repositories are safe (AFAIK).
-
Save File Corruption fix? PLEASE HELP
>The following is my educated guess at what to do next and I am a week-end warrior programmer at best. If someone with more experience comments, I would listen to them over me< If you have no programming experience, this may be a lost cause. It looks like the save file has a header area followed by a list of "zlib" compressed chunks. zlib.net may have a program that can open the save file and let you see the raw information inside it. Something after the header is garbled and it is preventing Save-Editors from decompressing that chunk OR from parsing what is inside of it. If you are lucky, the garbled bit will jump out at you and you can repair it.
-
Zlib Critical Vulnerability
These appears to be the relevant changes:
2022-07-30: https://github.com/madler/zlib/commit/eff308af425b67093bab25...
2022-08-08: https://github.com/madler/zlib/commit/1eb7682f845ac9e9bf9ae3...
The second commit definitely fixed a null pointer dereference, I am not sure if the CVE is referencing something else that was fixed by the first commit.
-
Gzip and Brotli Compression Level Estimator
It is actually possible to estimate the compression level without actually compressing everything again, because different levels use different strategies which can be identified in some cases. zlib in particular has three strategies [1] and preflate [2] leverages this to store the deflate stream reconstruction data without much bits.
[1] https://github.com/madler/zlib/blob/21767c6/deflate.c#L134-L...
[2] https://github.com/deus-libri/preflate/blob/master/preflate_...
-
What does it take to be a good programmer?
It's still a heap of old school C spaghetti https://github.com/madler/zlib/commit/eff308af425b67093bab25...
What are some alternatives?
LZ4 - Extremely Fast Compression algorithm
Snappy - A fast compressor/decompressor
LZMA - (Unofficial) Git mirror of LZMA SDK releases
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
brotli - Brotli compression format
haproxy - HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
Onion - C library to create simple HTTP servers and Web Applications.
Minizip-ng - Fork of the popular zip manipulation library found in the zlib distribution.