Solution for hash-map with >100M values

1. Chronicle Map

   1 5 2,856 7.8 Java

   Replicate your Key Value Store across your network, with consistency, persistance and performance.

   

   I've wrangled data sets in the ~600gb range using nothing but plain old Java and a few beefy boxes. This can all be kept in memory, but you have to go off-heap. You can use Chronicle Map and Chronicle Values to model this data and work with it off-heap in a way that's still very clean and object oriented. 128gb of RAM is cheap these days, whether you're in the cloud or not.

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. MapDB

   2 5 4,978 0.0 Java

   MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

   

   I have had good results with mapdb

4. Oak

   3 2 270 0.0 Java

   A Scalable Concurrent Key-Value Map for Big Data Analytics (by yahoo)

   

   Consider using an database (e.g. H2 embedded, redis) with an on-heap cache (e.g. Caffeine). Since you say it is a Zipfian distribution, the cache should absorb most of the requests. For an off-heap hashtable, you might try Oak as it is likely a faster implementation.

5. java-concurrent-hash-trie-map

   4 1 151 0.0 Java

   Java port of a concurrent trie hash map implementation from the Scala collections library

   

6. lasher

   5 1 4 0.0 Java

   Lasher is an embeddable key-value store written in Java.

   

   Do you need to update the data after initial load? If not, then I would suggest using my Paldb fork , otherwise you could try my lasher library. It's in early stage but first results are very promising, I was testing it with 10-100M elements and the performance was similar to java hashmap.

7. SmoothieMap

   6 1 301 0.0 Java

   A gulp of low latency Java

   

   Try https://github.com/TimeAndSpaceIO/SmoothieMap

Suggest a related project