file-system-access VS datasette-lite

You can read about the privacy concerts the community group published [1].

[1]: https://wicg.github.io/file-system-access/#privacy-considera...

WHATWG File System Standard provides a means to write directories and files to the private origin storage associated with a Web page origin. If you want you can use WICG File System Access API to write data directly to your filesystem in the browser. WHATWG File System Standard uses the same FileSystemDirectoryHandle and FileSystemFileHandle defined by File System Access API.

Chromium-based browsers support File Systeam Access API.

We have stopped using the file system access API: https://github.com/WICG/file-system-access/issues/401

Where file handling is concerned, there are at least

- File System Access API, https://wicg.github.io/file-system-access/

- File Handling, https://github.com/WICG/file-handling/blob/master/explainer....

- Origin Private File System, https://github.com/WICG/file-system-access/blob/main/AccessH...

There was also Storage Foundation API to which the reaction was "I don't think it's an acceptable outcome for the web platform to have that many ways to work with files" :) https://github.com/mozilla/standards-positions/issues/481 This one never saw the light of day.

Huh. Disregard what I wrote entirely, then. Reading through https://github.com/WICG/file-system-access/blob/main/AccessH..., I can see how they’ve bypassed most of the problems I saw—I was making unnecessary assumptions.

Thank you for correcting me.

> Browsers have never let anything (not even JS) have raw access to the host FS

I'm not sure what you mean by "raw access", but the File System Access API certainly allows web applications to do a lot of things.

> The File System Access API (formerly known as Native File System API and prior to that it was called Writeable Files API) enables developers to build powerful web apps that interact with files on the user's local device, like IDEs, photo and video editors, text editors, and more.

https://web.dev/file-system-access/

> After a user grants a web app access, this API allows the app to read or save changes directly to files and folders on the user’s device. Beyond reading and writing files, this API provides the ability to open a directory and enumerate its contents. Additionally, web apps can use this API to store references to files and directories they’ve been given access to, allowing the web apps to later regain access to the same content without requiring the user to select the same file again.

> Additionally this API also makes it possible for websites to get access to some directory without having to first prompt the user for access.

https://wicg.github.io/file-system-access/

It's not just a draft, it's been part of Chrome since version 78 in 2019.

> After a user grants access, this API allows web apps to read or save changes directly to files and folders on the user's device. It does all this by invoking the platform's own open and save dialog boxes.

https://blog.chromium.org/2019/09/chrome-78-beta-new-houdini...

Discussion at the time:

https://news.ycombinator.com/item?id=21032537

In the post, the Vite webserver is only used to serve the HTML and JS static files to the browser. When the user saves or opens text files, the code uses the web File System Access API (https://wicg.github.io/file-system-access/) to interact with the user file system.

Have you heard of Project Fugu and their idea of a File System Access API in the browser? https://wicg.github.io/file-system-access/

I don't know a C++ equivalent of HTML , prompt(), or File System Access API (https://wicg.github.io/file-system-access/; https://web.dev/file-system-access/) using JavaScript in the browser window.

Also see: https://github.com/simonw/datasette-lite

https://github.com/simonw/datasette-lite :

> You can use this tool to open any SQLite database file that is hosted online and served with a `access-control-allow-origin: ` CORS header. Files served by GitHub Pages automatically include this header, as do database files that have been published online using `datasette publish`.*

> [...] You can paste in the "raw" URL to a file, but Datasette Lite also has a shortcut: if you paste in the URL to a page on GitHub or a Gist it will automatically convert it to the "raw" URL for you

> To load a Parquet file, pass a URL to `?parquet=`

> [...] https://lite.datasette.io/?parquet=https://github.com/Terada...*

There are various *-to-sqlite utilities that load data into a SQLite database for use with e.g. datasette. E.g. Pandas with `dtype_backend='arrow'` saves to Parquet.

datasette plugins are written in Python and/or JS w/ pluggy:

Datasette Lite might be even better for this - you can construct URLs that link directly to examples: https://github.com/simonw/datasette-lite

There are some amazing things for SQLite in the browser especially if you're looking for ways to host queryable data for cheap.

I have a hacked up POC experimental version of datasette-lite to be able to look at multi-GB databases at https://github.com/simonw/datasette-lite/pull/49. It uses a hacked up chunk'd lazyFile implementation from emscripten and others to grap pages from Cloudflare R2.

It's a test with california's unclaimed property records (https://www.sco.ca.gov/upd_download_property_records.html) of a 28GB searching up that guy who owns Twitter: https://datasette-lite-lab.mindflakes.com/index.html?url=htt...

I think there may be a space for super-large multi-GB files served from static storage being accessible from SQlite as well. Another one would be this full-text search of a 43GB SQLite database of Wikipedia's full text search: http://static.wiki/ . Hearing there's official support for this is awesome and I hope they also might add some provisions for those sticking with POSIX/Emscripten as well.

I grafted the enhanced lazyFile implementation of this to datasette-lite relatively recently. Threw in a 18GB CSV from

https://www.sco.ca.gov/upd_download_property_records.html

into a FTS5 Sqlite Database which came out to about 28GB after processing:

POC, non-merging Draft PR for the hack:

https://github.com/simonw/datasette-lite/pull/49

You can run queries through it if you URL hack into it and just get to the query dialog, browsing is kind of a dud at the moment since datasette runs a count(*) which downloads everything.

Oh wow! I wonder how hard it would be to load that module into https://github.com/simonw/datasette-lite

datasette-lite – Datasette running in your browser using WebAssembly and Pyodide

I have an open issue for that here: https://github.com/simonw/datasette-lite/issues/28

My initial hunch is that this will be really difficult - probably require a fork of something like https://github.com/coleifer/pysqlite3 then compiled for WebAssembly.

I'm confident it's feasible, but I don't have the skills to figure it out myself.