What scientists must know about hardware to write fast code

• lzbench

  9 841 1.9 C

  lzbench is an in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

• SIMDCompressionAndIntersection

  1 410 2.0 C++

  A C++ library to compress and intersect sorted lists of integers using SIMD instructions

If you’re working with time series data and using a vectorized algorithm implemented with SIMD instructions you can achieve decompression speeds approaching 15 GB/s [1].

Anecdotally when working with biological signals at my day job, compression is a massive win and an absolute no brainer tradeoff when shuffling data across the network or even just storing in memory.

That said, I think your first point is still reasonable for certain types of data and compression algorithms.

[1] https://github.com/lemire/SIMDCompressionAndIntersection

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project