Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 12 data-compression Open-Source Projects
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
lzav
Fast In-Memory Data Compression Algorithm (inline C/C++) 460+MB/s compress, 2500+MB/s decompress, ratio% better than LZ4, Snappy, and Zstd@-1
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
Turbo-Range-Coder
TurboRC - Fastest Range Coder + Arithmetic Coding / Fastest Asymmetric Numeral Systems
-
wbz
A parallel implementation of the bzip2 data compressor in python, this data compression pipeline is using algorithms like Burrows–Wheeler transform (BWT) and Move to front (MTF) to improve the Huffman compression. For now, this tool only will be focused on compressing .csv files, and other files on tabular format.
Project mention: Zstd Content-Encoding planned to ship with Chrome 123 | news.ycombinator.com | 2024-02-07I'm still unconvinced about this addition. And I don't even dislike Zstandard.
The main motivation seems to be that while Zstandard is worse than Brotli at the highest level, it's substantially faster than Brotli when data has to be compressed on the fly with a limited computation budget. That might be true, but I'm yet to see any concrete or even anecdotal evidence even in the issue tracker [1] while there exist some benchmarks where both Zstandard and Brotli are fast enough for the web usage even at lower levels [2].
According to their FAQ [3] Meta and Akamai have successfully used Zstandard in their internal network, but my gut feeling is that they never actually tried to optimize Brotli instead. In fact, Meta employs the main author of Zstandard so it would have been easier to tune Zstandard instead of Brotli. While Brotli has some fundamental difference from Zstandard (in particular Brotli doesn't use arithmetic-equivalent coding), no one has concretely demonstrated that difference would prevent Brotli from being fast enough for dynamic contents in my opinion.
[1] https://issues.chromium.org/issues/40196713
[2] https://github.com/powturbo/TurboBench/issues/43
[3] https://docs.google.com/document/d/14dbzMpsYPfkefAJos124uPrl...
Project mention: LZAV – Fast In-Memory Data Compression Algorithm (In C) | news.ycombinator.com | 2023-07-19
I'm working on a game for Amiga (another 68k-based platform) and settled on ZX0 to decompress assets on the fly: https://github.com/einar-saukas/ZX0
I was originally using LZ4, but I switched to ZX0 after learning that it can do in-place decompression, which means I don't have to allocate separate memory for the compressed data. I'm very happy with the compression ratio, and decompression of large assets (~48kb) only takes a few frames on a 7MHz 68000.
Also of note is LZ4W, included in Sega Genesis Dev Kit (and discussed in the comments section of OP's article), a variant of LZ4 that only uses word-aligned operations. That makes it much faster on the 68000, which can struggle to efficiently handle 8-bit data. More info here: https://github.com/Stephane-D/SGDK/blob/master/bin/lz4w.txt
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
data-compression related posts
-
FC8 – Faster 68K Decompression (2016)
-
Zstd Content-Encoding planned to ship with Chrome 123
-
Ebiggers/libdeflate: Heavily optimized DEFLATE/zlib/gzip library
-
TurboBench: Dynamic/Static web content compression benchmark
-
TurboBench: Dynamic/Static web content compression benchmark
-
TurboBench: Dynamic/Static web content compression benchmark
-
Variation on RLE to Achieve Lossless Compression for Tabular Data
-
A note from our sponsor - InfluxDB
www.influxdata.com | 15 May 2024
Index
What are some of the best open-source data-compression projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | fpng | 830 |
| 2 | compression | 827 |
| 3 | secure-ls | 637 |
| 4 | TurboBench | 314 |
| 5 | lzav | 292 |
| 6 | bandwidth-hero | 267 |
| 7 | ancient | 204 |
| 8 | ZX0 | 148 |
| 9 | Turbo-Range-Coder | 63 |
| 10 | arcode-rs | 23 |
| 11 | wbz | 13 |
| 12 | arithmetic-coding | 12 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com