scrape-hacker-news-by-domain VS diaryman

Yeah I have a bunch of these using pretty-printed JSON - here's one that scrapes Hacker News for mentions of my site, for example: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time.

I think it's fine to use the term "scraping" to refer to downloading a JSON file.

These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML.

I do run Git scrapers that process HTML as well. A couple of examples:

scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository.

scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

I have a neat Hacker News scraping setup that I'm really pleased with.

The problem: I want to know when content from one of my sites is submitted to Hacker News, and keep track of the points and comments over time. I also want to be alerted when it happens.

Solution: https://github.com/simonw/scrape-hacker-news-by-domain/

This repo does a LOT of things.

It's an implementation of my Git scraping pattern - https://simonwillison.net/2020/Oct/9/git-scraping/ - in that it runs a script once an hour to check for more content.

It scrapes https://news.ycombinator.com/from?site=simonwillison.net (scraping the HTML because this particular feature isn't supported by the Hacker News API) using shot-scraper - a tool I built for command-line browser automation: https://shot-scraper.datasette.io/

The scraper works by running this JavaScript against the page and recording the resulting JSON to the Git repository: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

That solves the "monitor and record any changes" bit.

But... I want alerts when my content shows up.

I solve that using three more tools I built: https://datasette.io/ and https://datasette.io/plugins/datasette-atom and https://datasette.cloud/

This script here runs to push the latest scraped JSON to my SQLite database hosted using my in-development SaaS platform, Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

I defined this SQL view https://simon.datasette.cloud/data/hacker_news_posts_atom which shows the latest data in the format required by the datasette-atom plugin.

Which means I can subscribe to the resulting Atom feed (add .atom to that URL) in NetNewsWire and get alerted when my content shows up on Hacker News!

I wrote a bit more about how this all works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

I'm really pleased with the Hacker News scraping demo in this - it's an extension of the scraper I wrote back in March, using shot-scraper to execute JavaScript in headless Chrome and write the resulting JSON back to a Git repo: https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

My new demo also then pipes that data up to Datasette using curl -X POST - this script here: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

Shameless plug of basically the same idea except as a pip package:

https://github.com/Aperocky/diarycli

`pip install diarycli`

Alternatively there is a shell version if you are averse to python/pip package manager as well:

https://github.com/Aperocky/diaryman/blob/master/diaryman.sh

I've been using this for years, it doesn't have any fancy features - it only reliably opens up/create today's diary in vim whenever I type `diary` in command line. with some minor utilities.

Shameless plug on the same subject if you are vim user fond of terminal:

https://github.com/Aperocky/diarycli

`pip install diarycli`

Alternatively there is a shell version if you are averse to python/pip package manager as well:

https://github.com/Aperocky/diaryman/blob/master/diaryman.sh

The only way I can get myself to write things down is to have it one commands away in the terminal.

Diary script.

`$ diary` create/open a file for today's diary in vim: https://github.com/Aperocky/diaryman/blob/master/diaryman.sh

Or try `pip install diarycli`: https://github.com/Aperocky/diarycli, for a pip packaged python version that does the exact same thing.

I've actually kept diary and work logs, things I did not know I was capable of.

Having had good usage and some positive feedback on https://github.com/Aperocky/diaryman/blob/master/diaryman.sh, I decided to pack it into a pip package.

Essentially, all this does is to make creating diary super simple in the CLI, and resulting diary organized in a nice /diary/year/month/date.md hierarchy.

Having used this for a few years, I find this tool indispensable - I was never able to write diary consistently but once it was available via `diary` it became nature to utilize this to manage daily tasks at work and write personal reflections at home.

> For example, my oldest files were made in Microsoft Word on an iMac G3 running Mac OS 9. I can open them in a modern word processor, and they look similar – but it’s not the same. Some of the fonts and graphics are missing, and I don’t know where I’d find replacements.

> It’s even harder for an undocumented side project I abandoned years ago. Having the code isn’t the same as a working application.

The author's solution to this is apparently screenshots, I have to respectfully disagree.

For software, side project or not, it should probably come with dependency configurations (granted, in early 2000s this isn't as mature as it is today) and some tests. My side projects basically all have tests, these tests are vital for picking up years later and for validation while developing.

For personal notes, I use this script which upon `$ diary` would create/open an entry for the current day in the appropriate folder with vim: https://github.com/Aperocky/diaryman/blob/master/diaryman.sh. Text files will last forever, it has some basic flavoring with markdown, but that's it. The folder where this is indexed is without a doubt the most valuable data on my computer, and it stretches back years.

I do occasionally take screenshots but never for reasons that author find screenshot to be useful for.

For the unix fundamentalist out here:

https://github.com/Aperocky/diaryman/blob/master/diaryman.sh

I had this little script aliased from shell, whenever I type `diary` it creates/open the current days' note file in vim.

Here's my note taking system: https://github.com/Aperocky/diaryman/blob/master/diaryman.sh

Whenever I type `diary` in the terminal, it opens vim on a text file that corresponds to today. All of the diary files are nicely ordered in directory structure that goes $DIARY_ROOT/$year/$month/$day.md

It worked very well for me over years.