sqlitefs VS datasette

[3] is in Go, lacks a LICENSE file, has been inactive four months, the README is in Russian, but well the demo code works. pkg.go.dev will not display its API.

[1] https://github.com/andyhebear/sqlitefs

- https://github.com/simonw/datasette: An open source multi-tool for exploring and publishing data

15. Datasette: Explore and publish data with this open-source multi-tool - https://datasette.io/

I've been researching methods for exposing large data sets to users of my website, and came across the Atlassian Data Lake (https://www.atlassian.com/platform/analytics/what-is-atlassian-data-lake), and was curious - is this a thing that companies are doing now?

For personal projects and smaller scale non-dynamic data sets I've used Datasette (https://datasette.io/), but haven't seen anyone providing anything at scale beyond either an API for accessing data or some sort of periodic file export.

If anyone doubts that simonw knows Datasette well, I encourage a close examination of this URL: https://github.com/simonw/datasette

I have a neat Hacker News scraping setup that I'm really pleased with.

The problem: I want to know when content from one of my sites is submitted to Hacker News, and keep track of the points and comments over time. I also want to be alerted when it happens.

Solution: https://github.com/simonw/scrape-hacker-news-by-domain/

This repo does a LOT of things.

It's an implementation of my Git scraping pattern - https://simonwillison.net/2020/Oct/9/git-scraping/ - in that it runs a script once an hour to check for more content.

It scrapes https://news.ycombinator.com/from?site=simonwillison.net (scraping the HTML because this particular feature isn't supported by the Hacker News API) using shot-scraper - a tool I built for command-line browser automation: https://shot-scraper.datasette.io/

The scraper works by running this JavaScript against the page and recording the resulting JSON to the Git repository: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

That solves the "monitor and record any changes" bit.

But... I want alerts when my content shows up.

I solve that using three more tools I built: https://datasette.io/ and https://datasette.io/plugins/datasette-atom and https://datasette.cloud/

This script here runs to push the latest scraped JSON to my SQLite database hosted using my in-development SaaS platform, Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

I defined this SQL view https://simon.datasette.cloud/data/hacker_news_posts_atom which shows the latest data in the format required by the datasette-atom plugin.

Which means I can subscribe to the resulting Atom feed (add .atom to that URL) in NetNewsWire and get alerted when my content shows up on Hacker News!

I wrote a bit more about how this all works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

Datasette

Datasette is remarkably well-suited to exactly the use case you've described. You essentially just point it at a SQLite database and get a reasonably friendly interface for querying, sorting, all that good stuff. You can customize pretty much everything about it depending on how dirty you wanna get your hands, but the out-of-the-box experience might even suffice for what you're trying to do.

datasette? https://datasette.io/