mvsqlite
blueboat
Our great sponsors
mvsqlite | blueboat | |
---|---|---|
26 | 18 | |
1,293 | 1,918 | |
- | - | |
0.0 | 0.0 | |
8 days ago | about 1 year ago | |
Rust | Rust | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mvsqlite
-
FoundationDB: A Distributed Key-Value Store
I’ve been using FDB for toy projects for a while. It’s truly rock solid. That being said, I wish there were more layers.
Ideally someone could implement the firestore or dynamodb api on top.
-
Go bindings to SQLite using Wazero
For the rough plan, it's Cloud Backed SQLite meets FoundationDB.
-
SQLite-based databases on the Postgres protocol? Yes we can
- Oh, and if you're wondering about backup to S3, they have that too: https://github.com/libsql/bottomless
- Uh, sqld can integrated with this https://github.com/losfair/mvsqlite, so now your SQLite is backed by FoundationDB!?
- Meanwhile Litestream exists https://github.com/benbjohnson/litestream/
-
We Built Fly Postgres
This was on HN a few months back: https://github.com/losfair/mvsqlite
While not Spanner, it is essentially an open source db like AlloyDB or Aurora, pushing replication and scale out to the storage layer (in this case via FoundationDB). The most interesting bit of mvsqlite is it's multi-writer capabilities, using FoundationDB to perform page-level locks.
I'm neither the creator nor using it in production, but I'd love to see more DBs using FoundationDB as storage. It's a pretty cool solution.
-
Litestream doesn't do SQLite replication anymore (LiteFS does)
Shameless plug of my [mvSQLite](https://github.com/losfair/mvsqlite) project here! It's basically another distributed SQLite, but with support for everything expected from a proper distributed database: synchronous replication, strictly serializable transactions, + scalable reads and writes w/ multiple concurrent writers.
-
SQLite: QEMU All over Again?
The approach to edge-ifying SQLite taken in [1] looks quite promising - using FoundationDb as the storage handles a lot of hairiness.
This project looks really exciting!
I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version...
A few features on my wishlist:
1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction.
2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages.
3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2].
-
Show HN: Query SQLite files stored in S3
That DynamoDB VFS looks cool! I agree that the VFS api makes one think about plenty of crazy ideas. Someone is working on a VFS based on Foundation DB[0] that looks very promising. It was recently discussed here[1]
-
Turning SQLite into a Distributed Database
Hi mrkurt!
Litestream/LiteFS are amazing projects. The FUSE-based approach is interesting (I'm implementing something similar in mvSQLite, thanks for the idea!)
> Graceful failure
mvSQLite is designed to continue to operate under degraded network (there is a fault-injection test specifically for checking this property: https://github.com/losfair/mvsqlite/blob/1dd1a80d2ff7263b07a...). Network errors and service unavailability are handled with idempotent retries and not exposed to the application.
> Good for caching
mvSQLite caches pages read and written, and does differential cache invalidation (only remotely modified pages are invalidated in the local page cache). The local cache is just a regular KV store with invalidation strategies, and can be moved onto the disk. So it essentially becomes a consistent local database snapshot.
blueboat
-
What's with All the Runtimes for JavaScript?
Right now it is an exciting time for JavaScript. We just got a new shiny fast runtime Bun, with the last new kid Deno being released only 4 years ago, and we have edge computing/serverless runtimes like Cloudflare worker and Blueboat. With all these hypes for the JavaScript community, I could not help but ask, how come only JavaScript gets all these fancy new runtimes? Why don’t we hear these more often in other languages?
-
Show HN: Distributed SQLite on FoundationDB
Hello HN! I'm building mvsqlite, a distributed variant of SQLite with MVCC transactions, that runs on FoundationDB. It is a drop-in replacement that just needs an `LD_PRELOAD` for existing applications using SQLite.
I made this because [Blueboat](https://github.com/losfair/blueboat) needs a native SQL interface to persistent data. Apparently, just providing a transactional key-value store isn’t enough - it is more easy and efficient to build complex business logic on an SQL database, and it seems necessary to bring a self-hostable distributed SQL DB onto the platform. Since FoundationDB is Blueboat’s only stateful external dependency, I decided to build the SQL capabilities on top of it.
At its core, mvsqlite’s storage engine, mvstore, is a multi-version page store built on FoundationDB. It addresses the duration and size limits (5 secs, 10 MB) of FDB transactions, by handling multi-versioning itself. Pages are fully versioned, so they are always snapshot-readable in the future. An SQLite transaction fetches the read version during `BEGIN TRANSACTION`, and this version is used as the per-page range scan upper bound in future page read requests.
For writes, pages are first written to a content-addressed store keyed by the page's hash. At commit, hashes of each written page in the SQLite transaction is written to the page index in a single FDB transaction to preserve atomicity. With 8K pages and ~60B per key-value entry in the page index, each SQLite transaction can be as large as 1.3 GB (compared to FDB's native txn size limit of 10 MB).
mvsqlite is not yet "production-ready", since it hasn’t received enough testing, and I may still have a few changes to make to the on-disk format. But please ask here if you have any questions!
-
Show HN: Blueboat is an all-in-one, multi-tenant serverless JavaScript runtime
This sounds quite a bit like Cloudflare Workers, and they have a comparison page - https://github.com/losfair/blueboat/wiki/Comparison-with-Clo....
-
Are V8 isolates the future of computing?
Blueboat may be what you’re looking for
> If one writes Go or Rust, there are much better ways to run them than targeting WASM
wasm has its place, especially for contained workloads that can be wrapped in its strict capability boundaries (think, file-encoding jobs that shouldn't access anything else but said files: https://news.ycombinator.com/item?id=29112713).
> Containers are still the defacto standard.
wasmedge [0], atmo [1], krustlet [2], blueboat [3] and numerous other projects are turning up the heat [4]!
[0] https://github.com/WasmEdge/WasmEdge
[1] https://github.com/suborbital/atmo
[2] https://github.com/krustlet/krustlet
- Blueboat, an open-source alternative to Cloudflare Workers
-
Deno Deploy Beta 2
https://github.com/losfair/rusty-workers
They're not perfectly isolated to a high security standard such that you could deploy your own v8 workers SaaS. And they do have quirks and development woes. I haven't tested in production but if it's just your trusted apps wanting to exceed the cloudflare workers 30 scripts limit then both are wonderfully powerful solutions to put behind a https proxy.
What are some alternatives?
bun - Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one
awesome-sqlite - A curated list of awesome things related to SQLite
dqlite - Embeddable, replicated and fault-tolerant SQL engine.
litefs - FUSE-based file system for replicating SQLite databases across a cluster of machines
rqlite - The lightweight, distributed relational database built on SQLite.
jose - JWA, JWS, JWE, JWT, JWK, JWKS for Node.js, Browser, Cloudflare Workers, Deno, Bun, and other Web-interoperable runtimes.
deploy_feedback - For reporting issues with Deno Deploy
datasette-stripe - A web SQL interface to your Stripe account using Datasette.
workers-sdk - ⛅️ Home to Wrangler, the CLI for Cloudflare Workers®
fdb-document-layer - A document data model on FoundationDB, implementing MongoDB® wire protocol
SSVM - WasmEdge is a lightweight, high-performance, and extensible WebAssembly runtime for cloud native, edge, and decentralized applications. It powers serverless apps, embedded functions, microservices, smart contracts, and IoT devices.
litestream - Streaming replication for SQLite.