lzbench
SIMDCompressionAndIntersection
Our great sponsors
lzbench | SIMDCompressionAndIntersection | |
---|---|---|
9 | 1 | |
841 | 410 | |
- | - | |
1.9 | 2.0 | |
about 1 month ago | 10 months ago | |
C | C++ | |
- | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
lzbench
-
Chrome Feature: ZSTD Content-Encoding
For a benchmark on a standard set: https://github.com/inikep/lzbench/blob/master/lzbench18_sort...
-
My experience with btrfs so far
Do not re-compress your file into level 3. The decompression speed is largely the same between level 3 and 8, so you just wasting CPU doing nothing and making your files larger. See the bottom of the README: https://github.com/inikep/lzbench
-
Rsyncing 20TB locally
You can crunch the numbers yourself with this: https://github.com/inikep/lzbench
-
Lizard – efficient compression with fast decompression
Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1], which is more or less up-to-date.
[1] https://github.com/inikep/lzbench
- What scientists must know about hardware to write fast code
-
Zip-Ada development on LZMA compression
u/zertillon, maybe you could use lzbench, so you could compare it with a lot of other compression libraries. The problem is that it requires including the library in a single executable, so it might be more difficult to integrate than a C library (the benchmark is in C++).
-
Is there any site that lists the current SOTA for lossless compression?
Still updated: https://github.com/inikep/lzbench
-
will ZSTD impact L2ARC performance?
If you want to know the size a VM will compress to,. Zstd can be installed on any machine, so you can experiment easily. You can even run the benchmark https://github.com/inikep/lzbench
-
Save disk space for your games: BTRFS filesystem compression as alternative to CompactGUI on Linux
Are you sure about that? That's not what I see on https://github.com/inikep/lzbench and I tried to run that myself, although I have no idea which lzo to try so I went with what seemed the fastest...
SIMDCompressionAndIntersection
-
What scientists must know about hardware to write fast code
If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1].
Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory.
That said, I think your first point is still reasonable for certain types of data and compression algorithms.
[1] https://github.com/lemire/SIMDCompressionAndIntersection
What are some alternatives?
7-Zip-zstd - 7-Zip with support for Brotli, Fast-LZMA2, Lizard, LZ4, LZ5 and Zstandard
xsimd - C++ wrappers for SIMD intrinsics and parallelized, optimized mathematical functions (SSE, AVX, AVX512, NEON, SVE))
CompactGUI - Transparently compress active games and programs using Windows 10/11 APIs [Moved to: https://github.com/IridiumIO/CompactGUI]
TurboPFor - Fastest Integer Compression
CompactGUI - Transparently compress active games and programs using Windows 10/11 APIs
std_find_simd - std::find simd version
11Zip - Dead simple zipping / unzipping C++ Lib
Vc - SIMD Vector Classes for C++
qemu
highway - Performance-portable, length-agnostic SIMD with runtime dispatch
ffi-overhead - comparing the c ffi (foreign function interface) overhead on various programming languages
zip-ada - Zip-Ada: a standalone, portable Ada library for .zip archives. Includes LZMA byte stream encoder & decoder pair.