Solution for hash-map with >100M values

1. Chronicle Map

   1 5 2,967 8.2 Java

   Replicate your Key Value Store across your network, with consistency, persistance and performance.

   

   I've wrangled data sets in the ~600gb range using nothing but plain old Java and a few beefy boxes. This can all be kept in memory, but you have to go off-heap. You can use Chronicle Map and Chronicle Values to model this data and work with it off-heap in a way that's still very clean and object oriented. 128gb of RAM is cheap these days, whether you're in the cloud or not.

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. MapDB

   2 5 5,047 0.0 Java

   MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine.

   

   I have had good results with mapdb

4. Oak

   3 2 276 0.0 Java

   A Scalable Concurrent Key-Value Map for Big Data Analytics (by yahoo)

   

   Consider using an database (e.g. H2 embedded, redis) with an on-heap cache (e.g. Caffeine). Since you say it is a Zipfian distribution, the cache should absorb most of the requests. For an off-heap hashtable, you might try Oak as it is likely a faster implementation.

5. java-concurrent-hash-trie-map

   4 1 155 0.0 Java

   Java port of a concurrent trie hash map implementation from the Scala collections library

   

6. lasher

   5 1 4 0.0 Java

   Lasher is an embeddable key-value store written in Java.

   

   Do you need to update the data after initial load? If not, then I would suggest using my Paldb fork , otherwise you could try my lasher library. It's in early stage but first results are very promising, I was testing it with 10-100M elements and the performance was similar to java hashmap.

7. SmoothieMap

   6 1 307 0.0 Java

   A gulp of low latency Java

   

   Try https://github.com/TimeAndSpaceIO/SmoothieMap

Suggest a related project