Embedded write-heavy on-disk cache, write-amplification

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/golang

Our great sponsors
  • Scout APM - Less time debugging, more time building
  • SonarLint - Deliver Cleaner and Safer Code - Right in Your IDE of Choice!
  • SaaSHub - Software Alternatives and Reviews
  • squirrel

    Go sqlite3 cache (by anacrolix)

    I'm looking for an embedded database/KV-store that supports a write-heavy workload of large blocks of bytes and some kind of eviction policy. I'm currently using sqlite3 with a bunch of triggers and the blob API, but it's not really suitable for write-heavy workloads. I've currently exposed the interface somewhat in https://github.com/anacrolix/squirrel, the primary use case is from https://github.com/anacrolix/torrent. My recent research suggests an LSM-based KV-store like rocksdb or leveldb, but those don't have great interfaces in Go, and don't seem to support an eviction policy as far as I can tell (which is surprising given they would be very well suited to it). There are some alternatives like buntdb, but those all look designed for smaller/string values.

  • torrent

    Full-featured BitTorrent client package and utilities

    I'm looking for an embedded database/KV-store that supports a write-heavy workload of large blocks of bytes and some kind of eviction policy. I'm currently using sqlite3 with a bunch of triggers and the blob API, but it's not really suitable for write-heavy workloads. I've currently exposed the interface somewhat in https://github.com/anacrolix/squirrel, the primary use case is from https://github.com/anacrolix/torrent. My recent research suggests an LSM-based KV-store like rocksdb or leveldb, but those don't have great interfaces in Go, and don't seem to support an eviction policy as far as I can tell (which is surprising given they would be very well suited to it). There are some alternatives like buntdb, but those all look designed for smaller/string values.

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • goleveldb

    LevelDB key/value database in Go.

    We're using go-leveldb for a reasonably high-load case here at my $dayjob.

  • badger

    Fast key-value DB in Go.

    At the time we have selected it, we have compared it with BoltDB but that completely failed to fit the bill as it had abysmal write performance. I recall we've also compared it with BadgerDB, but at the time the latter was in its infancy so we had to rule it out.\ These days I'd recommend to give it a close look.

  • diskv

    A disk-backed key-value store.

  • missinggo

    Stuff that's missing in Go stdlib, or hasn't made it into its own repo.

    I wrote something similar to diskv with caching. I stopped using it due to OS file overhead: https://github.com/anacrolix/missinggo/tree/master/filecache

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts