sqlite VS redimo.go

> You can get substantially better performance out of sqlite by using the lower level https://github.com/crawshaw/sqlite, turning on WAL etc, using a connection per goroutine for reads, and sending batches of writes over a buffered channel / queue to a dedicated writer thread. That way you can turn off SQLite’s built in per-connection mutex but still be thread safe since each connection is only used on a single thread at a time.

Would this protect against a row update in the middle of a read? e.g. would a row at least be internally consistent at the time it was read?

The default go sqlite driver is https://github.com/mattn/go-sqlite3, which is quite lovely, but I ran into issues with concurrency on read only databases.

I'm now using https://github.com/crawshaw/sqlite and it seems to address those issues (but I haven't gotten around to setting up a proper test to confirm). It may be worth perusing if you do run into performance problems. It does come with the caveat of not being a database/sql driver though.

crawshaw/sqlite - Low-level Go interface to SQLite

crawshaw/sqlite actually uses CGo -- it's a wrapper around the C version of SQLite. For example, see https://github.com/crawshaw/sqlite/blob/23d646f8ac00d9dd2390...

zombiezen/go-sqlite uses cznic's pure Go converted version of SQLite, so avoids CGo. It's explicitly stated to be "a fork of crawshaw.io/sqlite that uses modernc.org/sqlite, a CGo-free SQLite package. It aims to be a mostly drop-in replacement for crawshaw.io/sqlite."

The second well known driver is https://github.com/crawshaw/sqlite

Did this for DynamoDB over the pandemic. Helped me learn both the Redis API and DynamoDB. https://github.com/dbProjectRED/redimo.go

Want to do the same thing with Postgres as well.

The Redis API and data structures are really nice and have been tuned to be useful in a wide variety of situations. Implementing these APIs is different concurrent, compute and storage models is a nice way to learn them and get options open.

The way that I learnt the ins and outs of DynamoDB (and there is a lot to learn if you want to use it effectively) is by implementing all the Redis data structures and commands on it. That helped understand both systems in one shot.

The key concept in Dynamo is that you use a partition key on all your bits of data (my mental model is that you get one server per partition) and you then can arrange data using a sort key in that partition. You can then range/inequality query over the sort keys. That’s the gist of it.

The power and scalability comes from the fact that each partition can be individually allocated and scaled, so as long as you spread over partitions you have practically no limits.

And you can do quite a bit with that sort key range/inequality thing. I was pleasantly surprised by how much of Redis I could implement: https://github.com/dbProjectRED/redimo.go

Running a Fly app backed by DynamoDB Global Tables is an option. DDB keeps a copy of your data in all the regions you specify, each Fly instance can connect to the nearest region, and writes are propagated with eventual consistency & last write wins.

And most Redis commands can be mapped to DDB, I worked on a lib to do that.

https://github.com/dbProjectRED/redimo.go

https://github.com/sudhirj/aws-regions.go

https://aws.amazon.com/dynamodb/global-tables/

Redimo this open source project from Sudhir Jonathan that provides a library that allows you to use the Redis API on DynamoDB and bridges the two and translates the Redis API operations into space / time / cost-efficient DynamoDB API calls. Redimo is especially well suited to serverless environments, since there is no pool of connections to handle and DynamoDB is purpose-built for near-zero management use. Read the detailed README.md file to understand the nuances of this project.