sqlitefs VS datasette

> but also presents as a true filesystem.

As does:

https://github.com/guardianproject/libsqlfs

https://github.com/narumatt/sqlitefs

(I know nothing about these, just got them from a quick search)

If anyone doubts that simonw knows Datasette well, I encourage a close examination of this URL: https://github.com/simonw/datasette

I have a neat Hacker News scraping setup that I'm really pleased with.

The problem: I want to know when content from one of my sites is submitted to Hacker News, and keep track of the points and comments over time. I also want to be alerted when it happens.

Solution: https://github.com/simonw/scrape-hacker-news-by-domain/

This repo does a LOT of things.

It's an implementation of my Git scraping pattern - https://simonwillison.net/2020/Oct/9/git-scraping/ - in that it runs a script once an hour to check for more content.

It scrapes https://news.ycombinator.com/from?site=simonwillison.net (scraping the HTML because this particular feature isn't supported by the Hacker News API) using shot-scraper - a tool I built for command-line browser automation: https://shot-scraper.datasette.io/

The scraper works by running this JavaScript against the page and recording the resulting JSON to the Git repository: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

That solves the "monitor and record any changes" bit.

But... I want alerts when my content shows up.

I solve that using three more tools I built: https://datasette.io/ and https://datasette.io/plugins/datasette-atom and https://datasette.cloud/

This script here runs to push the latest scraped JSON to my SQLite database hosted using my in-development SaaS platform, Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

I defined this SQL view https://simon.datasette.cloud/data/hacker_news_posts_atom which shows the latest data in the format required by the datasette-atom plugin.

Which means I can subscribe to the resulting Atom feed (add .atom to that URL) in NetNewsWire and get alerted when my content shows up on Hacker News!

I wrote a bit more about how this all works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

Datasette

Datasette is remarkably well-suited to exactly the use case you've described. You essentially just point it at a SQLite database and get a reasonably friendly interface for querying, sorting, all that good stuff. You can customize pretty much everything about it depending on how dirty you wanna get your hands, but the out-of-the-box experience might even suffice for what you're trying to do.

datasette? https://datasette.io/

Would this tool work for you? Look like specifically tailored to exposing DB data as APIs: https://datasette.io/

Very cool. Have you considered making the data available as a (static) Datasette website? https://datasette.io/ You can probably host it on Github pages for free.

My project https://datasette.io/ is ideal for this kind of thing. You can use https://sqlite-utils.datasette.io/ to load JSON data into a SQLite database, then publish it with Datasette.

I'm definitely keen on suggestions for improvements I can make to the default UI.

Datasette provides both a JSON API (easily enabled for CORS access) and supports custom templates, so it's possible to customize the UI any way you like.

So far I've not seen many examples of extensive customization. I use the custom templates a lot myself - these four sites are all just regular Datasette with some custom templates:

- https://datasette.io/

- https://til.simonwillison.net/

- https://www.niche-museums.com/

- https://www.rockybeaches.com/us/pillar-point

Source code is on GitHub for all four.