edgesearch VS sql.js

From https://westurner.github.io/hnlog/#comment-30603322 re https://en.wikipedia-on-ipfs.org/wiki/Database_index :

https://github.com/wilsonzlin/edgesearch :

> * Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps *

> "Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers"

WASM or [C++] to WASM?

TIL about Roaring Bitmaps: /?q=roaring+bitmap https://medium.com/@amit.desai03/roaring-bitmaps-fast-data-s...

https://en.wikipedia-on-ipfs.org/wiki/Database_index

https://news.ycombinator.com/item?id=28021766 re: edgesearch, HTTP/3 (QUIC (UDP))

> Serverless full-text search with Cloudflare Workers, WebAssembly, and Roaring Bitmaps https://github.com/wilsonzlin/edgesearch

>> How it works: Edgesearch builds a reverse index by mapping terms to a compressed bit set (using Roaring Bitmaps) of IDs of documents containing the term, and creates a custom worker script and data to upload to Cloudflare Workers

I've also tried full-text-search in worker by pre-indexing the content, works very fast even with a JS engine - less than 5ms to make a search in 5MB of text.

It runs out of CPU-time at 6MB of text though.

There's someone that made a WASM for the same thing too, it's definitely faster and can handle a bit more text.

https://github.com/wilsonzlin/edgesearch

Hi HN! A friend and I were inspired by projects like https://github.com/sql-js/sql.js and the idea of querying files served over CDN with HTTP range requests. We started thinking: what would a database that was specifically designed for this type of use case look like? So we started building one, and we landed on a functional prototype that we're pretty proud of!

With our prototype, Appendable, we're able to serve and query large (GB+) datasets by hosting them on a static file host like Amazon S3 or Cloudflare R2 without running a separate server and worrying about things like tail latency, replication, and connection pooling -- all that is handled for us by the file hoster.

Additionally, one tenet that we have been following is Appendable won't touch your underlying data, so your jsonl file is preserved and we point at that data instead of consuming it into an Appendable-specific file format. This keeps your data yours and makes it easy to introspect the data: just open it up with your favorite editor aka vim.

We're curious what you think, we're excited to build this out further to get the performance even better and add features like pubsub. Everything is open source at https://github.com/kevmo314/appendable.

Kevin and Matthew

My project is powered by sql.js, I recommend checking that out if you're interested - https://github.com/sql-js/sql.js/

Maybe you can try to use some SQL database compiled as Web Assembly Modules? Like this one for example: https://github.com/sql-js/sql.js

If you want to have persistence, then I would go with a database like Dexie, as it uses IndexedDB and has transactions. If you just want something that's in memory, you could look at Sql.js or something simple like lowdb.

not necessarily. you can host an html/js/sqlite site on github pages for free. json -> sqlite3 js -> sql

Link: https://sql.js.org

Does it work with really large files? Like, >100mb or so. I was considering making something similar but with sqlite.js [1], but the problem with it is that it loads everything in memory, so I wasn't entirely sure how it will deal with larger workloads.

[1]: https://sql.js.org/#/