Embedded write-heavy on-disk cache, write-amplification

• squirrel

  2 18 8.1 Go

  Go sqlite3 cache (by anacrolix)

I'm looking for an embedded database/KV-store that supports a write-heavy workload of large blocks of bytes and some kind of eviction policy. I'm currently using sqlite3 with a bunch of triggers and the blob API, but it's not really suitable for write-heavy workloads. I've currently exposed the interface somewhat in https://github.com/anacrolix/squirrel, the primary use case is from https://github.com/anacrolix/torrent. My recent research suggests an LSM-based KV-store like rocksdb or leveldb, but those don't have great interfaces in Go, and don't seem to support an eviction policy as far as I can tell (which is surprising given they would be very well suited to it). There are some alternatives like buntdb, but those all look designed for smaller/string values.

• torrent

  22 5,313 9.3 Go

  Full-featured BitTorrent client package and utilities

I'm looking for an embedded database/KV-store that supports a write-heavy workload of large blocks of bytes and some kind of eviction policy. I'm currently using sqlite3 with a bunch of triggers and the blob API, but it's not really suitable for write-heavy workloads. I've currently exposed the interface somewhat in https://github.com/anacrolix/squirrel, the primary use case is from https://github.com/anacrolix/torrent. My recent research suggests an LSM-based KV-store like rocksdb or leveldb, but those don't have great interfaces in Go, and don't seem to support an eviction policy as far as I can tell (which is surprising given they would be very well suited to it). There are some alternatives like buntdb, but those all look designed for smaller/string values.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• goleveldb

  15 6,018 0.0 Go

  LevelDB key/value database in Go.

We're using go-leveldb for a reasonably high-load case here at my $dayjob.

• badger

  30 13,347 7.0 Go

  Fast key-value DB in Go.

  

At the time we have selected it, we have compared it with BoltDB but that completely failed to fit the bill as it had abysmal write performance. I recall we've also compared it with BadgerDB, but at the time the latter was in its infancy so we had to rule it out.\ These days I'd recommend to give it a close look.

• diskv

  3 1,324 0.0 Go

  A disk-backed key-value store.

• missinggo

  1 83 3.2 Go

  Stuff that's missing in Go stdlib, or hasn't made it into its own repo.

I wrote something similar to diskv with caching. I stopped using it due to OS file overhead: https://github.com/anacrolix/missinggo/tree/master/filecache

Suggest a related project