Sparkey is a simple constant key/value storage library

• sparkey

  1 1,149 4.0 C

  Simple constant key/value storage library, for read-heavy systems with infrequent large bulk inserts.

  

• civboot

  10 326 0.0

  Civilizational Bootstraper: landing page and wiki

  

Hey, this is effectively the same idea I'm going to implement!

https://github.com/civboot/civboot/blob/main/blog/0012-dev-l...

Mine will be slightly more than key/value. My index file will support indexes of arbitrary record fields as well as the index of all items in loam, but it's effectively the same idea.

Note you could keep a live instance running to handle live writes, which uses an in-memory hash for recent writes and updates the hash index as-needed. Because the index can be rebuilt from the data there is no concern about data loss.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• hammerspace

  2 364 10.0 Ruby

  Off-heap large object storage

  

At Airbnb we implementing translation serving on top of Sparkey; part of our Ruby system open sourced here: https://github.com/airbnb/hammerspace

The way the system worked (iirc it’s been a while): we’d periodically dump all the translation strings into a sparkey S3 path, and use a Kubernetes daemonset to fetch the sparkey file to each host in the cluster. Finally application pods written in any language use a host volume to read translation strings from the local sparkey file. Certainly overkill for smaller apps but with a bunch of services and hundreds of thousands of translation strings, the out of band updates are well worth it and keep the app container small.

• pogreb

  4 1,224 5.3 Go

  Embedded key-value store for read-heavy workloads written in Go

• sparkey-java

  1 117 5.3 Java

  Java implementation of the Sparkey key value store

  

Now when I'm at an actual computer, I see that it was implemented, at least in the Java version. Removing that made for a 30% speed increase: https://github.com/spotify/sparkey-java/pull/65

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project