mvsqlite VS litefs

- Oh, and if you're wondering about backup to S3, they have that too: https://github.com/libsql/bottomless

- Uh, sqld can integrated with this https://github.com/losfair/mvsqlite, so now your SQLite is backed by FoundationDB!?

- Meanwhile Litestream exists https://github.com/benbjohnson/litestream/

This was on HN a few months back: https://github.com/losfair/mvsqlite

While not Spanner, it is essentially an open source db like AlloyDB or Aurora, pushing replication and scale out to the storage layer (in this case via FoundationDB). The most interesting bit of mvsqlite is it's multi-writer capabilities, using FoundationDB to perform page-level locks.

I'm neither the creator nor using it in production, but I'd love to see more DBs using FoundationDB as storage. It's a pretty cool solution.

Shameless plug of my [mvSQLite](https://github.com/losfair/mvsqlite) project here! It's basically another distributed SQLite, but with support for everything expected from a proper distributed database: synchronous replication, strictly serializable transactions, + scalable reads and writes w/ multiple concurrent writers.

The approach to edge-ifying SQLite taken in [1] looks quite promising - using FoundationDb as the storage handles a lot of hairiness.

1: https://github.com/losfair/mvsqlite

This project looks really exciting!

I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version...

A few features on my wishlist:

1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction.

2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages.

3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2].

[1] https://github.com/losfair/mvsqlite

[2] https://univalence.me/posts/mvsqlite-bench-20220930

That DynamoDB VFS looks cool! I agree that the VFS api makes one think about plenty of crazy ideas. Someone is working on a VFS based on Foundation DB[0] that looks very promising. It was recently discussed here[1]

[0]: https://github.com/losfair/mvsqlite

[1]: https://news.ycombinator.com/item?id=32269287

Hi mrkurt!

Litestream/LiteFS are amazing projects. The FUSE-based approach is interesting (I'm implementing something similar in mvSQLite, thanks for the idea!)

> Graceful failure

mvSQLite is designed to continue to operate under degraded network (there is a fault-injection test specifically for checking this property: https://github.com/losfair/mvsqlite/blob/1dd1a80d2ff7263b07a...). Network errors and service unavailability are handled with idempotent retries and not exposed to the application.

> Good for caching

mvSQLite caches pages read and written, and does differential cache invalidation (only remotely modified pages are invalidated in the local page cache). The local cache is just a regular KV store with invalidation strategies, and can be moved onto the disk. So it essentially becomes a consistent local database snapshot.

Came over this article as I was looking for interesting resources in the SQLite ecosystem. I'm building mvsqlite (https://github.com/losfair/mvsqlite), as an attempt to turn SQLite into a proper distributed (not just replicated) database. Check it out if you are looking for this kind of stuff!

There's a page in the wiki doing the comparison: https://github.com/losfair/mvsqlite/wiki/Comparison-with-dql...

- Ben is developing https://github.com/superfly/litefs at Fly after stopping the streaming replication effort https://github.com/benbjohnson/litestream/issues/8

- And, of course, SQLite has announced a new backend that hopes to support concurrent writes and streaming replication: https://sqlite.org/hctree/doc/hctree/doc/hctree/index.html

What a time for SQLite

I wonder if there's any chance of synergy with this project https://github.com/superfly/litefs

Litestream is one-way replication, so it can't sync across multiple writers.

LiteFS is Ben Johnson's newer project that does SQLite synchronization across nodes: https://github.com/superfly/litefs

> If a LiteFS node is way behind on replication, can it let the app know when it's caught up?

LiteFS exposes the replication position via a file[1] suffixed with "-pos" for each database. That has a monotonically increasing TXID that you could compare to the primary.

We track latency via a Prometheus metric but we don't currently expose that easily to the application. I added an issue for that[2].

> have one Fly instance running and another in cold standby. On deploy, the cold instance boots, fails the health check until replication catches up, then takes over while the first instance goes cold.

When LiteFS starts up, it waits for the node to either connect to the primary and catch up or become the primary itself before it starts the user's application (if running in a supervisor mode). That would probably get you most of the way there to what you're trying to do.

[1]: https://fly.io/docs/litefs/position/

[2]: https://github.com/superfly/litefs/issues/253

Well there are SQLite replicators like https://github.com/superfly/litefs but just don't put the code with the SQLite in the lambda. You can still have it talk to the other binary. I'm not well versed on sharing resources between lambdas and the app.

It's not great. tv42 (who maintains the FUSE implementation we use) commented on it recently[1]. macOS support will be via a VFS extension in the future.

[1]: https://github.com/superfly/litefs/issues/119#issuecomment-1...

LiteFS author here. You can also set up LiteFS to have a single, static leader instead of using Consul if you don't want that dependency. I need to write up some docs but there's some info on the PR itself: https://github.com/superfly/litefs/pull/47

We actually kinda see parts of this in https://github.com/superfly/litefs, albeit for SQLite.

And here we see some ideas forming around "pluggable storage for PostgresQL": https://wiki.postgresql.org/wiki/Future_of_storage#Pluggable...

Seriously! If any of this sounds interesting to build, reach out, and we'll make it happen!

Looks like LiteFS invalidates the cached file contents on every update of position. This means there is never stale data to be read.

(The client needs to make sure to consume that in a single pread/read syscall, or it could observe a sheared state.)

Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse

https://github.com/superfly/litefs/blob/52e269d4b04070690ce2...

The TH3 test is proprietary and, IIRC, it might only be available to SQLite Consortium members and that's $125k/year. We do pay for SQLite support at Fly.io but not quite to that level.

We do have plans to run against the Tcl test suite[1] although most of that test suite is not applicable to LiteFS since it tests higher level constructs.

Since LiteFS acts on the raw pages, it really just functions similar to a VFS. Out of the 130K source lines of code in SQLite, only 4.8k are for the Unix VFS. As such, the testing coverage from the SQLite test suite mostly tests non-VFS code.

[1]: https://github.com/superfly/litefs/issues/17