InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises. Learn more →
Top 23 C Compression Projects
Zstandard - Fast real-time compression algorithmProject mention: Float Compression 3: Filters | news.ycombinator.com | 2023-02-01
Interesting to match with the observations from the practice of using ClickHouse for time series:
1. Reordering to SOA helps a lot - this is the whole point of column-oriented databases.
2. Specialized codecs like Gorilla, DoubleDelta, and FPC lose to simply using ZSTD compression in most cases, both in compression ratio and in performance.
3. Specialized time-series DBMS like InfluxDB or TimescaleDB lose to general-purpose relational OLAP DBMS like ClickHouse .
 https://arxiv.org/pdf/2204.09795.pdf "SciTS: A Benchmark for Time-Series Databases in Scientific Experiments and Industrial Internet of Things" (2022)
Brotli compression formatProject mention: Framer Update: 2x Faster Sites | reddit.com/r/framer | 2023-02-02
We serve your site from a global cache location close to your visitors to make sure your site loads fast. In addition, we use an advanced HTML and text compression algorithm called Brotli. Compressed content is now cached, so we can send it directly to your visitors instead of compressing each request individually. In our tests this often improves loading speed by up to 2x, which will have a very positive impact on your Lighthouse scores like LCP. This will be especially noticeable on larger sites, so you can scale your site without worry.
Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.
Extremely Fast Compression algorithmProject mention: Micron Unveils 24GB and 48GB DDR5 Memory Modules | AMD EXPO and Intel XMP 3.0 compatible | reddit.com/r/gadgets | 2023-01-21
Yeah, sure, when you have monster core counts. on regular systems, not so much, here's from their own github page. it achieves, eh, 5GB/s on memory to memory transfers, i.e. best case scenario. so, uh, no? i'm not even sure it's any better than the CPU decompressor one Nvidia used.
A massively spiffy yet delicately unobtrusive compression library.Project mention: Quite OK Image is now my favorite asset format | news.ycombinator.com | 2022-12-18
A minimal viable Deflate decompressor is not exactly complex, although slower than mainline zlib.
Collection of cross-platform one-file C/C++ libraries with no dependencies, primarily used for gamesProject mention: How many colors are too many colors for Windows Terminal? | news.ycombinator.com | 2022-05-14
It's a simple and relatively straightforward approach that a sufficiently bright programmer would come up in their own while looking at the design constraints though, so overall I find it a bit meaningless to find the ultimate person for the "original idea".
LZFSE compression library and command line tool
Columnar storage extension for Postgres built as a foreign data wrapper. Check out https://github.com/citusdata/citus for a modernized columnar storage implementation built as a table access method.
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
Modern audio compression for the internet.Project mention: Multiple tags with the same name in metadata | reddit.com/r/ffmpeg | 2023-01-08
If there are multiple tags with the same name, Ffmpeg will only use the last tag. If you really need to have multiple tags with the same name in your OPUS files, use opusenc instead (https://opus-codec.org/). Beware that some playback software does not display multiple artists gracefully.
zlib replacement with optimizations for "next generation" systems.Project mention: Zlib Critical Vulnerability | news.ycombinator.com | 2022-10-14
Zlib-ng doesn't contain the same code, but it appears that their equivalent inflate() when used with their inflateGetHeader() implementation was affected by a similar problem: https://github.com/zlib-ng/zlib-ng/pull/1328
Also similarly, most client code will be unaffected because `state->head` will be NULL, because they (most client code) won't have used inflateGetHeader() at all.
New generation entropy codecs : Finite State Entropy and Huff0Project mention: Silly Lossy Text Compression Idea | news.ycombinator.com | 2022-05-21
Sounds similar to: https://github.com/Cyan4973/FiniteStateEntropy
> The modern data compression is mainly based on two approaches to entropy coding: Huffman (HC) and arithmetic/range coding (AC). The former is much faster, but approximates probabilities with powers of 2, usually leading to relatively low compression rates. The latter uses nearly exact probabilities - easily approaching theoretical compression rate limit (Shannon entropy), but at cost of much larger computational cost.
A portable, simple zip library written in C (by kuba--)Project mention: Don't Use RAR | news.ycombinator.com | 2022-02-25
Small strings compression library
Fork of the popular zip manipulation library found in the zlib distribution.
A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`.Project mention: WASM compression benchmarks and the cost of missing compression APIs | news.ycombinator.com | 2023-02-02
Related to compressing data before storing on SSD:
Blosc - faster than memcpy()
On right circumstances Blosc is so fast that even speed ups reading data from RAM (read less, decompress in L1 and L2 caches)
lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressorsProject mention: Lizard – efficient compression with fast decompression | news.ycombinator.com | 2022-05-24
Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench , which is more or less up-to-date.
Fastest Integer CompressionProject mention: q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before | reddit.com/r/rust | 2022-02-17
I'm the author of TurboPFor-Integer-Compression. Q_compress is a very interresting project, unfortunatelly it's difficult to compare it to other algorithms. There is not binary or test data files (with q_compress results) available for a simple benchmark. Speed comparison would also be helpfull.
A new p7zip fork with additional codecs and improvements (forked from https://sourceforge.net/projects/sevenzip/ AND https://sourceforge.net/projects/p7zip/).
Lizard (formerly LZ5) is an efficient compressor with very fast decompression. It achieves compression ratio that is comparable to zip/zlib and zstd/brotli (at low and medium compression levels) at decompression speed of 1000 MB/s and faster. (by inikep)Project mention: Lizard – efficient compression with fast decompression | news.ycombinator.com | 2022-05-24
A better and stronger spiritual successor to BZip2.Project mention: In need of crazy compression | reddit.com/r/compression | 2022-10-15
bzip3 : https://github.com/kspalaiologos/bzip3
ZSON is a PostgreSQL extension for transparent JSONB compressionProject mention: JSONB and storage | reddit.com/r/PostgreSQL | 2023-01-04
Have you considered ZSON? It requires you to compile a dictionary based on sample data - but given you're expecting most of your data to fit one of two variations this could be quite space efficient.
tools to create and extract Squashfs filesystemsProject mention: How to get Kali tools and Snap on a Chromebook | dev.to | 2022-03-10
You need to install a few dependencies, notably the SquashFS and FUSE, and Snap itself:
A simple C library for compressing lists of integers using binary packing
Quick serialization of R objects (by traversc)Project mention: How to remove all columns except a select few with data.table? | reddit.com/r/Rlanguage | 2022-10-12
My initial rec was to use the fst package, but I read the other thread and you said it's not compatible. qs works in a similar way / with similar performances and should be compatible with any R version > 3.0.2, but I have personally never used it.
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
C Compression related posts
WASM compression benchmarks and the cost of missing compression APIs
2 projects | news.ycombinator.com | 2 Feb 2023
Framer Update: 2x Faster Sites
1 project | reddit.com/r/framer | 2 Feb 2023
Float Compression 3: Filters
3 projects | news.ycombinator.com | 1 Feb 2023
Micron Unveils 24GB and 48GB DDR5 Memory Modules | AMD EXPO and Intel XMP 3.0 compatible
1 project | reddit.com/r/gadgets | 21 Jan 2023
Weekly r/Tattoos Question/FreeTalk Thread! - January 14, 2023
1 project | reddit.com/r/tattoos | 21 Jan 2023
We're wasting money by only supporting gzip for raw DNA files
6 projects | news.ycombinator.com | 9 Jan 2023
Multiple tags with the same name in metadata
2 projects | reddit.com/r/ffmpeg | 8 Jan 2023
A note from our sponsor - InfluxDB
www.influxdata.com | 8 Feb 2023
What are some of the best open-source Compression projects in C? This list will help you: