SaaSHub helps you find the best software and product alternatives Learn more →
Pcodec Alternatives
Similar projects and alternatives to pcodec
-
gdal
GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
-
flamegraph
Easy flamegraphs for Rust projects and everything else, without Perl or pipes <3 (by flamegraph-rs)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
Turbo-Range-Coder
TurboRC - Fastest Range Coder + Arithmetic Coding / Fastest Asymmetric Numeral Systems
-
-
-
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better pcodec alternative or higher similarity.
pcodec reviews and mentions
Posts with mentions or reviews of pcodec.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2023-04-22.
-
Learnings from making things fast
Context: I've been iterating on my side project pcodec (a codec for columns of numerical data) and have gradually improved decompression speed from ~150MB/s to ~1GB/s. Not everything here is novel or Rust-specific, but here's what I've learned in the process:
- Compressing bytes?
-
Worries about tANS?
For context: I'm creating an experimental successor to my library Quantile Compression, which does good compression for numerical sequences and has several users. I have a variable number of symbols which may be as high as 212 in some cases, but is ~26 in most cases. The data is typically 216 to 224 tokens long.
-
Quantile Compression, a compression format for numerical data that improves compression ratio by ~30% over alternatives
I'm not a member, but you can use the CLI to try it out pretty easily: https://github.com/mwlon/quantile-compression/tree/main/q_compress_cli . Let me know how it does
-
I built Quantile Compression, which could make all our numerical columnar data 25% smaller.
You can try it out very easily with the CLI which works on CSV and Parquet columns now, e.g. cargo run --release compress --csv my.csv --col-name my_column out.qco
-
Quantile Compression: 35% higher compression ratio for numeric sequences than any other compressor
Right, please don't try to use it for general files. It looks like zpaq is kinda hard to set up except on windows, so I'm probably not going to, but I encourage you to try it out! There's an example you can use to generate a bunch of random numerical distributions, outputting binary files, .qco, and other formats.
- Q_compress: Lossless compressor and decompressor for numerical data
-
q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before
Here's how you can generate benchmark data, including binary files: https://github.com/mwlon/quantile-compression/blob/main/q_compress/examples/primary.md
-
Quantile Compression, a format and algorithm for numerical sequences offering 35% higher compression ratio than .zstd.parquet.
I made a simple CLI for compressing and inspecting .qco files. Not available on package managers yet, but it's still pretty easy to try out: https://github.com/mwlon/quantile-compression/blob/main/CLI.md
- Quantile Compression (q-compress), a new compression format and rust library that shrinks real-world columns of numerical data 10-40% smaller than other methods
-
A note from our sponsor - SaaSHub
www.saashub.com | 1 May 2024
Stats
Basic pcodec repo stats
19
245
8.8
7 days ago
mwlon/pcodec is an open source project licensed under Apache License 2.0 which is an OSI approved license.
The primary programming language of pcodec is Rust.
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com