queensland-traffic-conditions VS scrape-hacker-news-by-domain

I've been promoting this idea for a few years now, and I've seen an increasing number of people put it into action.

A fun way to track how people are using this is with the git-scraping topic on GitHub:

https://github.com/topics/git-scraping?o=desc&s=updated

That page orders repos tagged git-scraping by most-recently-updated, which shows which scrapers have run most recently.

As I write this, just in the last minute repos that updated include:

https://github.com/drzax/queensland-traffic-conditions

https://github.com/jasoncartwright/bbcrss

https://github.com/jackharrhy/metrobus-timetrack-history

https://github.com/outages/bchydro-outages

Yeah I have a bunch of these using pretty-printed JSON - here's one that scrapes Hacker News for mentions of my site, for example: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time.

I think it's fine to use the term "scraping" to refer to downloading a JSON file.

These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML.

I do run Git scrapers that process HTML as well. A couple of examples:

scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository.

scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

I'm really pleased with the Hacker News scraping demo in this - it's an extension of the scraper I wrote back in March, using shot-scraper to execute JavaScript in headless Chrome and write the resulting JSON back to a Git repo: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

My new demo also then pipes that data up to Datasette using curl -X POST - this script here: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...