How should I handle storing and reading from large amounts of data in my project?

This page summarizes the projects mentioned and recommended in the original post on /r/rust
Data structures C# Ruby Cplusplus Python

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • Apache Avro

    Apache Avro is a data serialization system.

    • Maybe it will be simpler to serialise all the data in a more compact data format, such as avro (its readme is in here), a row based format that seems to be able to use zstd/bzip/xz.

  • flatdata

    Write-once, read-many, minimal overhead binary structured file format.

    • Shameless self-plug: You could have a look at https://github.com/heremaps/flatdata to see if it fits your needs: Read-only memory mapped flat data structures providing fast random access.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts