-
tinyStats
Statistics about data (cardinality estimation, frequent item detection, approximate counting,...)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I wrote a gui for it, tot help deal with many files being in various states: https://github.com/brenthuisman/par2deep
Another option is to use an IBLT (Inverted Bloom Lookup Table). it is easy to implement. I recently wrote a file repair tool that uses it: https://github.com/thomasmueller/tinyStats/blob/master/src/m... (not production ready)
This isn't a turn key utility, but it's relatively easy to call from Go. It allows you to set the number of data shards and parity shards, it's pretty fast. It's at
https://github.com/klauspost/reedsolomon
Related posts
-
Show HN: Hacker News over SSH – Browse HN Articles Directly from Your Terminal
-
Ask HN: How do you develop and maintain a good note-taking habit?
-
Rabbit R1 can be run on a Android device
-
Flags Are Not Languages
-
Download your Learn course content with this free and open-source tool. All you need is a working computer and basic Python knowledge, and you can save a local copy of your Learn courses' content for future reference after the end of the term.