django-readers VS datasette-lite

* For read endpoints and associated business logic, I'd encourage https://www.django-readers.org/ (disclaimer: I'm the author).

We solved the N+1 queries problem where I work by raising the level of abstraction from "queries plus serialisation" to "what shape data is required". We open sourced the solution at https://www.django-readers.org/.

django-readers – function-oriented toolkit for better organisation of business logic and efficient selection and projection of data in Django projects

Also see: https://github.com/simonw/datasette-lite

https://github.com/simonw/datasette-lite :

> You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*

> [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you

> To load a Parquet file, pass a URL to `?parquet=`

> [...] https://lite.datasette.io/?parquet=https://github.com/Terada...*

There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.

datasette plugins are written in Python and/or JS w/ pluggy:

Datasette Lite might be even better for this - you can construct URLs that link directly to examples: https://github.com/simonw/datasette-lite

There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap.

I have a hacked up POC experimental version of datasette-lite to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grap pages from Cloudflare R2.

It's a test with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter: https://datasette-lite-lab.mindflakes.com/index.html?url=htt...

I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search: http://static.wiki/ . Hearing there's official support for this is awesome and I hope they also might add some provisions for those sticking with POSIX/Emscripten as well.

I grafted the enhanced lazyFile implementation of this to datasette-lite relatively recently. Threw in a 18GB CSV from

https://www.sco.ca.gov/upd_download_property_records.html

into a FTS5 Sqlite Database which came out to about 28GB after processing:

POC, non-merging Draft PR for the hack:

https://github.com/simonw/datasette-lite/pull/49

You can run queries through it if you URL hack into it and just get to the query dialog, browsing is kind of a dud at the moment since datasette runs a count(*) which downloads everything.

Oh wow! I wonder how hard it would be to load that module into https://github.com/simonw/datasette-lite

datasette-lite – Datasette running in your browser using WebAssembly and Pyodide

I have an open issue for that here: https://github.com/simonw/datasette-lite/issues/28

My initial hunch is that this will be really difficult - probably require a fork of something like https://github.com/coleifer/pysqlite3 then compiled for WebAssembly.

I'm confident it's feasible, but I don't have the skills to figure it out myself.