edgesearch VS dqlite

From https://westurner.github.io/hnlog/#comment-30603322 re https://en.wikipedia-on-ipfs.org/wiki/Database_index :

https://github.com/wilsonzlin/edgesearch :

> * Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps *

> "Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers"

WASM or [C++] to WASM?

TIL about Roaring Bitmaps: /?q=roaring+bitmap https://medium.com/@amit.desai03/roaring-bitmaps-fast-data-s...

https://en.wikipedia-on-ipfs.org/wiki/Database_index

https://news.ycombinator.com/item?id=28021766 re: edgesearch, HTTP/3 (QUIC (UDP))

> Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch

>> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers

I've also tried full-text-search in worker by pre-indexing the content, works very fast even with a JS engine - less than 5ms to make a search in 5MB of text.

It runs out of CPU-time at 6MB of text though.

There's someone that made a WASM for the same thing too, it's definitely faster and can handle a bit more text.

https://github.com/wilsonzlin/edgesearch

If you're interested in this, here are some related projects that all take slightly different approaches:

- LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io

- dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io

- cr-sqlite is a Rust-based loadable extension that adds CRDT changeset generation and reconciliation to SQLite: https://github.com/vlcn-io/cr-sqlite

Slightly related but not really (no multi writer, no C-level SQLite API or other restrictions):

- comdb2 (Bloombergs multi-homed RDMS using SQLite as the frontend)

- rqlite: RDMS with HTTP API and SQLite as the storage engine, used for replication and strong consistency (does not scale writes)

- litestream/LiteFS: disaster recovery replication

- liteserver: active read-only replication (predecessor of LiteSync)

I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite

Anyone knows how the DB this is about, https://rqlite.io/, compares with https://dqlite.io/ by Canonical (both seem to be distributed versions of sqlite)?

For folks' context, the new tool that's being discussed in the thread mentioned by the parent here is litefs [0], as well as which you can also look at rqlite [1] and dqlite [2], which all provide different trade-offs (e.g. rqlite is 'more strongly consistent' than litefs).

[0]: https://github.com/superfly/litefs

[1]: https://github.com/rqlite/rqlite

[2]: https://github.com/canonical/dqlite

I presume you're familiar with https://github.com/canonical/dqlite (made by my employer) and https://github.com/rqlite/rqlite (unrelated)? How will mvsqlite compare to those?

Check out dqlite, it's sqlite but with a raft consensus to distribute changes through a log: https://dqlite.io/ You can link it in as a library too, it sounds like exactly what you want.