Looking for fast, space-efficient key-lookup

• yottaStore

  9 82 1.8

  A datastore aiming at linear scalability up to the yottabyte range. Inspired by dynamo and cassandra.

  

  I copied this approach from several papers, with some improvements, for my datastore.

• CodeRabbit

  coderabbit.ai featured

  CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

  

• goleveldb

  15 6,145 0.0 Go

  LevelDB key/value database in Go.

  Looks like a job for GoLevelDB.

• bbolt

  19 8,182 9.0 Go

  An embedded key/value database for Go.

  

  - bbolt for storage on disk. In order to get the smallest db file size possible make sure you insert the keys in order and set:

• cuckoofilter

  1 1,118 2.1 Go

  Cuckoo Filter: Practically Better Than Bloom

  - a cuckoo filter for fast lookup. This has around a 3% false positive rate. There are other implementations however that have a much lower rate. You can store the filter in the database as well in a different bucket so you don't have to rebuild the filter on startup.

• RoaringBitmap

  25 3,512 8.5 Java

  A better compressed bitset in Java: used by Apache Spark, Netflix Atlas, Apache Pinot, Tablesaw, and many others

  

  Use a two stage approach, with a bloom/cuckoo filter stored as a https://roaringbitmap.org/ in memory. Then a secondary key/value store on disk (bolt or anything else).

• diskv

  3 1,387 0.0 Go

  A disk-backed key-value store.

  https://github.com/peterbourgon/diskv might be a solution

• go

  2,158 123,364 10.0 Go

  The Go programming language

  

  Most hash map (or set) implementations also overallocate quite a bit to reduce the number of collisions. You could use a custom map implementation that has a tuned load factor, that way you can trade speed for memory. You can have a look at the go map implementation to see how that could work: https://github.com/golang/go/blob/master/src/runtime/map.go With that said, unless you have a very good reason to go down that rabbit hole I'd avoid it.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• pebble

  11 4,820 9.8 Go

  RocksDB/LevelDB inspired key-value database in Go

  

  https://github.com/cockroachdb/pebble Pure go SSD native key-value store. You could think of it as map[[]byte][]byte on persistent storage.

• cdb

  1 65 10.0 Go

  A native golang implementation of cdb (http://cr.yp.to/cdb.html) (by colinmarc)

  github.com/colinmarc/cdb is a Go implementation,

Suggest a related project