SIMDCompressionAndIntersection
TurboPFor
Our great sponsors
| SIMDCompressionAndIntersection | TurboPFor | |
|---|---|---|
| 1 | 8 | |
| 410 | 743 | |
| - | - | |
| 2.0 | 8.5 | |
| 10 months ago | about 2 months ago | |
| C++ | C | |
| Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
SIMDCompressionAndIntersection
-
What scientists must know about hardware to write fast code
If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1].
Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory.
That said, I think your first point is still reasonable for certain types of data and compression algorithms.
[1] https://github.com/lemire/SIMDCompressionAndIntersection
TurboPFor
- Show HN: Time Series Benchmark TurboPFor,TurboFloat,TurboFloat LzX,TurboGorilla
-
Show HN: HN:The Gorilla in the Room:Exploring RedisTimeSeries Optimizations
[4] https://github.com/powturbo/TurboPFor-Integer-Compression/is...
-
Float Compression 9: Lzsse and Lizard
The bytedelta described in the blog is suboptimal and does might not work with other datasets.
Download icapp from https://github.com/powturbo/TurboPFor-Integer-Compression/re... and make your own tests with your data.
[1] https://github.com/powturbo/TurboPFor-Integer-Compression
- Show HN: 1D/2D/3D Lossless/Lossy Floating Point Compression with TurboPFor
-
How do Games manage NPC schedules?
I use a fake database paired with compressed bits for flags and integer compression for various other traits. They follow a navigation guide similar to wind for foliage.
-
Quantile Compression: 35% higher compression ratio for numeric sequences than any other compressor
It could be nice to see a comparison against https://github.com/powturbo/TurboPFor-Integer-Compression !
-
q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before
I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.
-
C Deep
TurboPFor - Fastest integer compression. GPL-2.0-or-later
What are some alternatives?
lzbench - lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors
x3-rust - X3 Lossless Audio Compression for Rust
xsimd - C++ wrappers for SIMD intrinsics and parallelized, optimized mathematical functions (SSE, AVX, AVX512, NEON, SVE))
libuv - Cross-platform asynchronous I/O
std_find_simd - std::find simd version
42_CheatSheet - A comprehensive guide to 50 years of evolution of strict C programming, a tribute to Dennis Ritchie's language
Vc - SIMD Vector Classes for C++
CRoaring - Roaring bitmaps in C (and C++), with SIMD (AVX2, AVX-512 and NEON) optimizations: used by Apache Doris, ClickHouse, and StarRocks
highway - Performance-portable, length-agnostic SIMD with runtime dispatch
MessagePack - MessagePack serializer implementation for Java / msgpack.org[Java]
encoding - Integer Compression Libraries for Go
Allegro - The official Allegro 5 git repository. Pull requests welcome!