fusionauth-site VS shot-scraper

Have tried to use this a few times but it's a big lift. Like adding unit tests to an existing application, it's a good idea to start small. And vale doesn't make this super easy (at least I couldn't figure out an easy way to do this). If I could wave my wand, I'd want a way to:

* apply vale to just the doc I was working on

* have a minimal set of rules

* add to them over time

At $curjob, we have a detailed public list of rules of doc ( https://github.com/FusionAuth/fusionauth-site/blob/master/Do... ) and as our team expands, I'd love to have them be applied rigorously. vale seems like a good fit, but there's an activation energy that I haven't been able to get over yet.

I am not aware of any other cli tools similar to this, though, so totally admire the team behind it.

Thanks for the comments! Will try to address them in the near future, tracking here: https://github.com/FusionAuth/fusionauth-site/issues/2170

I actually have much more experience with plantuml (we actually have a plugin which generates diagrams for our jekyll/asciidoc docs site: https://github.com/FusionAuth/fusionauth-site/blob/master/si... ), but I have recently started using mermaid for a project and like it so far.

I had a colleague (thanks Sanjay!) make a similar tool at a hackfest (you can see the source code here: https://github.com/FusionAuth/fusionauth-site/blob/master/sr... ).

I've modified it a bit to take a URL, but haven't yet set it up to read a config file to make a large number of screenshots easy to do.

We do outline certain fields or other areas in the doc to highlight a point. That's caused some hesitation on my part. However, it looks like I could use imagemagick to automatically put a red box or similar on an image (with a `-draw` command).

We have a ton of screenshots (600+) throughout our doco, and a way to initialize our product to a known state, so the pieces are all there.

One of these days it'll be worthwhile to do this.

I had one of these recently! https://github.com/simonw/shot-scraper/pull/133/files

They're /incredibly/ rare though.

If anyone wants the raw data, it's available in window._Flourish_data variable on https://flo.uri.sh/visualisation/16818696/embed

Which means you can extract it with my https://shot-scraper.datasette.io/ tool like this:

    shot-scraper javascript \

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with.

I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable/javascript.html

This is exactly what the `Accept` HTTP header is for https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ac...

I think the author is generally correct that all JSON should be provided in a single request, but if you want to prove it, then you should be able to change your accept header to and from `application/json`/`text/html seeing nearly identical data.

In fact, this is what both GitLab and Github do. Try it out!

`curl -L https://github.com/simonw/shot-scraper` (text/html)

`curl --header "Accept: application/json" -L https://github.com/simonw/shot-scraper` (application/json)

Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time.

I think it's fine to use the term "scraping" to refer to downloading a JSON file.

These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML.

I do run Git scrapers that process HTML as well. A couple of examples:

scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository.

scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

You can use shot-scraper: https://github.com/simonw/shot-scraper

I have a neat Hacker News scraping setup that I'm really pleased with.

The problem: I want to know when content from one of my sites is submitted to Hacker News, and keep track of the points and comments over time. I also want to be alerted when it happens.

Solution: https://github.com/simonw/scrape-hacker-news-by-domain/

This repo does a LOT of things.

It's an implementation of my Git scraping pattern - https://simonwillison.net/2020/Oct/9/git-scraping/ - in that it runs a script once an hour to check for more content.

It scrapes https://news.ycombinator.com/from?site=simonwillison.net (scraping the HTML because this particular feature isn't supported by the Hacker News API) using shot-scraper - a tool I built for command-line browser automation: https://shot-scraper.datasette.io/

The scraper works by running this JavaScript against the page and recording the resulting JSON to the Git repository: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

That solves the "monitor and record any changes" bit.

But... I want alerts when my content shows up.

I solve that using three more tools I built: https://datasette.io/ and https://datasette.io/plugins/datasette-atom and https://datasette.cloud/

This script here runs to push the latest scraped JSON to my SQLite database hosted using my in-development SaaS platform, Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

I defined this SQL view https://simon.datasette.cloud/data/hacker_news_posts_atom which shows the latest data in the format required by the datasette-atom plugin.

Which means I can subscribe to the resulting Atom feed (add .atom to that URL) in NetNewsWire and get alerted when my content shows up on Hacker News!

I wrote a bit more about how this all works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

Sounds a lot like Simon Willison's open source project shot-scraper

https://github.com/simonw/shot-scraper