Our great sponsors
-
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
-
LevelDB
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
alasql
AlaSQL.js - JavaScript SQL database for browser and Node.js. Handles both traditional relational tables and nested JSON data (NoSQL). Export, store, and import data from localStorage, IndexedDB, or Excel.
This looks pretty efficient. Some chains can be interacted with without e.g. web3.js? LevelDB indexes aren't SQLite.
Datasette is one application for views of read-only SQLite dbs with out-of-band replication. https://github.com/simonw/datasette
There are a bunch of *-to-sqlite utilities in corresponding dogsheep project.
Arrow JS for browser client access to DuckDB might be faster but without full SQLite SQL compatibility. https://arrow.apache.org/docs/js/
https://duckdb.org/
> Direct Parquet & CSV querying
I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...
If you just do an occasional key/value lookup, you don't need 1.2 MiB of WebAssembly. [1]
I'd solve it via sharding: divide the database into N pieces via range- or mod-sharding. [1] Choose an N that's large enough for each piece to be reasonably small. When you look up a key, fetch the shard of interest.
You can put each piece into separate files (a little simpler to code, and most static servers will use pre-gzipped files for "Content-Encoding: gzip requests" easily, but you waste more disk space due to internal fragmentation) or one file (with range serving and an index of the byte range offset for each piece).
The format for each piece can be anything, eg json (simple) or an sstable-like format (more efficient). [3]
[1] Content-Length of https://phiresky.github.io/youtube-sponsorship-stats/sql-was...
[2] mod-sharding means: piece[i] has all the keys where hash(key) % N = i.
[3] https://github.com/google/leveldb/blob/master/doc/table_form...
Hi, I'm working on SkySQL, maybe it's useful for you: https://github.com/upshot-tech/SkySQL
I wrote a similar thing in Rust for a Factorio mod manager - mods are hosted on the remote HTTP server as ZIP files and the mod manager needs a single info.json file from the ZIP for the mod metadata, so it avoids downloading the whole mod and then unpacking it by building a file abstraction that uses HTTP range queries to download chunks. For ZIP files the directory is stored at the end at an unknown offset, so the read pattern is to gradually seek backwards from the end until you find the start of the directory, then find the file entry, then read the file.
I didn't fiddle with the window sizes like the TFA, but I did optimize it so that reading chunk N+1 of the file reused the response readed of chunk N rather than making a new request. Furthermore I keep an LRU cache of the last three chunks rather than keep all of them in memory, because the ZIP files are each only read once.
[1]: https://github.com/Arnavion/fac-rs/blob/master/src/solve/web...
[2]: https://github.com/Arnavion/fac-rs/blob/master/src/solve/zip...
For similar results, you can use SQL with IndexedDB running fully client-side: https://github.com/agershun/alasql