tailetc
go-memdb
Our great sponsors
| tailetc | go-memdb | |
|---|---|---|
| 2 | 3 | |
| 131 | 3,070 | |
| - | 1.7% | |
| 0.0 | 3.6 | |
| almost 2 years ago | 2 months ago | |
| Go | Go | |
| BSD 3-clause "New" or "Revised" License | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tailetc
-
Sched - In-process Go Job Scheduler. With Cron Support and Prometheus Metrics
https://github.com/tailscale/tailetc/blob/b2fa539c2383d30d03e0eea1052022af132dca9f/tailetc.go#L142
-
An Unlikely Database Migration
Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.)
In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd (https://github.com/tailscale/tailetc/blob/main/tailetc.go). That's quite a brave task considering how many databases fail at Jepsen. Have you tried running Jepsen tests on tailetc yourself? You also mentioned a secondary index system which I assume is built on top of tailetc again? How does that interact with tailetc?
Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten you quite far.
We don't really get the full overview from a short blog post like this though so maybe it turns out to be a great solution for you. The code quality itself looks great and it seems that you have thought about all of the hard problems.
go-memdb
-
I created an in-memory SQL database called MemSQL as a learning project
For another in-memory database example, you could also check out https://github.com/hashicorp/go-memdb, but it's based on a radix tree implementation.
-
Why use a real K/V database over a MapStore?
I just think its a glorified map. Yeah, for example go-memdb list some of its advantages in their https://github.com/hashicorp/go-memdb/blob/master/README.md README. But it is not clear whether you gain any performance gain over a map except for avoiding the locking as explained.
-
An Unlikely Database Migration
>So we invested what probably amounts to two or three weeks of engineering time into designing in-memory indexes that are transactionally consistent
I'll be interested to see that part if it does get open sourced. I've used https://github.com/hashicorp/go-memdb which sounds similar.
What are some alternatives?
etcd - Distributed reliable key-value store for the most critical data of a distributed system
dbbench - 🏋️ dbbench is a simple database benchmarking tool which supports several databases and own scripts
lungo - A MongoDB compatible embeddable database and toolkit for Go.
badger - Fast key-value DB in Go.
buntdb - BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
bolt
go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.
cockroach - CockroachDB - the open source, cloud-native distributed SQL database.
tidb - TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial
groupcache - groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.
eliasdb - EliasDB a graph-based database.
InfluxDB - Scalable datastore for metrics, events, and real-time analytics