vu128: Efficient variable-length integers

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • compressed-vec

    SIMD Floating point and integer compressed vector library

    If I’m following, this presents an alternative format and implementation to LEB128 which encodes and decodes substantially faster. Notably, the implementation is quite simple. Cool! And agreed that modern CPUs really suffer from branches.

    Should I interpret the plot to mean the average elapsed wall clock time per integer decoded/encoded? And can I conclude the throughput is the reciprocal? So about 100,000 integers per second or around a 1 GB/s of decompressed data.

    I know this is a bit unfair because the implementation is much more complex, but my first thought is why I would use vu128 instead of Lemire’s Stream VByte: https://arxiv.org/abs/1709.08990

    A slight tangent but I stumbled on this library which stores floats XOR’ed with the previous float in the stream: https://github.com/velvia/compressed-vec it seems really clever to me! They reference “Gorilla: A Fast, Scalable, In-Memory Time Series Database” which in turn references two 2006 papers: “Fast Lossless Compression of Scientific Floating-Point Data” and “Fast and Efficient Compression of Floating-Point Data”. Frustratingly, the FB paper doesn’t benchmark their XOR-based floating point encoding but the earlier two papers do.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • rust

    Empowering everyone to build reliable and efficient software.

    It seems to be more fussy about compiler optimizations, though: https://github.com/rust-lang/rust/issues/125543

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Designing a SIMD Algorithm from Scratch

    3 projects | news.ycombinator.com | 28 Nov 2023
  • lz4_flex 0.11: Gainzzzzz Unleashed!: Performance Improvements Detailed in Blogpost (LZ4 De/compression)

    3 projects | /r/rust | 18 Jun 2023
  • Filtering a Vector with AVX-2 & AVX-512 in Rust

    4 projects | /r/rust | 1 Sep 2022
  • Do you ever use unsafe { .. } when not implementing custom data structures or interacting with external C code?

    5 projects | /r/rust | 14 Aug 2022
  • Tolower() in Bulk at Speed

    5 projects | news.ycombinator.com | 27 Jun 2022

Did you konow that Rust is
the 5th most popular programming language
based on number of metions?