An Unlikely Database Migration

• pgtest

  1 50 5.3 Go

  PostgreSQL test helper for Go

For PostgreSQL and Go, here's a package to spin up a temp in-mem PostgreSQL: https://github.com/rubenv/pgtest/

• tailetc

  2 131 0.0 Go

  Discontinued total-memory-cache etcd v3 client

  

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.)

In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd (https://github.com/tailscale/tailetc/blob/main/tailetc.go). That's quite a brave task considering how many databases fail at Jepsen. Have you tried running Jepsen tests on tailetc yourself? You also mentioned a secondary index system which I assume is built on top of tailetc again? How does that interact with tailetc?

Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten you quite far.

We don't really get the full overview from a short blog post like this though so maybe it turns out to be a great solution for you. The code quality itself looks great and it seems that you have thought about all of the hard problems.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• go-memdb

  3 3,067 3.6 Go

  Golang in-memory database built on immutable radix trees

  

>So we invested what probably amounts to two or three weeks of engineering time into designing in-memory indexes that are transactionally consistent

I'll be interested to see that part if it does get open sourced. I've used https://github.com/hashicorp/go-memdb which sounds similar.

• lungo

  2 448 5.2 Go

  A MongoDB compatible embeddable database and toolkit for Go.

I found myself in a similar situation sometime ago with MongoDB. In one project my unit tests started slowing me down too much to be productive. In another, I had so little data that running a server alongside it was a waste of resources. I invested a couple of weeks in developing a SQLite type of library[1] for Go that implemented the official Go drivers API with a small wrapper to select between the two. Up until now, it paid huge dividends in both projects ongoing simplicity and was totally worth the investment.

[1]: https://github.com/256dpi/lungo

• homelab

  2 1,882 0.0

  Brad's homelab setup (by bradfitz)

We have a lot of Kubernetes experience on the team. Multiple of us run Kubernetes clusters in our home labs (mine: https://github.com/bradfitz/homelab), and one of us used to be on the Google GKE team as an SRE, and is the author of https://metallb.universe.tf/ (which multiple of us also use).

Us _not_ using Kubernetes isn't because we don't know how to use it. It's because we _do_ know how to use it and when _not_ to use it. :)

• SQLBoiler

  42 6,424 7.8 Go

  Generate a Go ORM tailored to your database schema.

  

GORM is nice for beginners because it's pretty easy to get started with, but in my experience starts to fall apart when you need to do anything more complex or start to scale up. I've had good experiences with SQLBoiler [0] on the other hand. I haven't used it in for anything in production, but it's been a breeze to use in a couple of personal projects, and it handles complex SQL queries much better compared to GORM.

0: https://github.com/volatiletech/sqlboiler

Suggest a related project