sqld VS mvsqlite

There's https://github.com/libsql/sqld , but sqlite's concurrency model doesn't always work well with long-lived transactions (and just the network hop can be slower than a local transaction), especially if you want to write.

For instructions on how to build sqld from source, see the docs.

I also discovered https://github.com/libsql/sqld/ which provides similar functionality ("Bottomless replication"), and exposes the database over the network via the Postgres protocol. I haven't played with this yet, but it would be neat to try running some apps that require Postgres with this.

There are some language-specific clients suited for data transaction with a Turso database, such as @libsql/client for TypeScript, libsql-client for Python, and libsql-client for Rust(currently works with Cloudflare workers), but for simplicity, we’re going to use curl scripts. We’ll interact with the database by sending HTTP POST requests containing the JSON-encoded SQL queries that we’d otherwise run on the SQL shell as already demonstrated. By supporting database interaction via the HTTP protocol, Turso guarantees easy access from all types of applications, especially edge functions.

- And you can embed it in your application and it will really talk to a "SQLite" database over the network: https://github.com/libsql/sqld (not a lot on consistency, I wonder how multiple writers are handled?)

I’ve been using FDB for toy projects for a while. It’s truly rock solid. That being said, I wish there were more layers.

Ideally someone could implement the firestore or dynamodb api on top.

https://github.com/losfair/mvsqlite

For the rough plan, it's Cloud Backed SQLite meets FoundationDB.

- Oh, and if you're wondering about backup to S3, they have that too: https://github.com/libsql/bottomless

- Uh, sqld can integrated with this https://github.com/losfair/mvsqlite, so now your SQLite is backed by FoundationDB!?

- Meanwhile Litestream exists https://github.com/benbjohnson/litestream/

Shameless plug of my [mvSQLite](https://github.com/losfair/mvsqlite) project here! It's basically another distributed SQLite, but with support for everything expected from a proper distributed database: synchronous replication, strictly serializable transactions, + scalable reads and writes w/ multiple concurrent writers.

This project looks really exciting!

I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version...

A few features on my wishlist:

1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction.

2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages.

3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2].

[1] https://github.com/losfair/mvsqlite

[2] https://univalence.me/posts/mvsqlite-bench-20220930

That DynamoDB VFS looks cool! I agree that the VFS api makes one think about plenty of crazy ideas. Someone is working on a VFS based on Foundation DB[0] that looks very promising. It was recently discussed here[1]

[0]: https://github.com/losfair/mvsqlite

[1]: https://news.ycombinator.com/item?id=32269287