map-of-github VS shot-scraper

You may appreciate the map of github [0] it's a fantastic piece of work that trended here a while back.

[0]: https://anvaka.github.io/map-of-github/

https://www.reddit.com/r/dataisbeautiful/comments/i8saks/ive...

https://www.reddit.com/r/dataisbeautiful/comments/ic2i0k/oc_...

Stock chart as a landscape image

And I also like these "races" basically a timelapse of a chart

https://www.reddit.com/r/dataisbeautiful/comments/f68fzr/oc_...

And whatever anvaka does, all can be tried out in the browser

https://github.com/anvaka?tab=repositories&q=&type=&language...

e.g. map of reddit:

https://github.com/anvaka/map-of-reddit

Video Demo: https://www.reddit.com/r/dataisbeautiful/comments/12pem68/oc...

Similarly map of github

https://github.com/anvaka/map-of-github

Or package manager visualizations:

https://github.com/anvaka/pm

Google search completion visualizations

https://github.com/anvaka/vs

I don't find these argument convincing

- hackers in Shenzhen are underrepresented b/c of the language barrier. There is actually a huge Chinese language ecosystem (a lot of it off Github on Gitee) that is sort of impenetrable but you come across it all the time. I just typically have no way to use their work b/c it's under-documented in English. If you zip around this map you will see lots of "Zh" islands: https://anvaka.github.io/map-of-github/

- While I understand that it's not as financial comfortable for a program in Indian compared wtih the US or Europe, Chinese and Russian hackers aren't significantly better off. Your typical Chinese program is also of course worried about working for a big company and making money and all of that jazz

- "escape the hell that is 'unpaid open source work' " - I think maybe is the crux of it... Most people work on open source work b/c they enjoy it.. I don't personally know anyone who got hired from a deliberate effort to build an open source portfolio. I wouldn't recommend it to anyone as a way to try to find a job

The source code with description of the method is available here: https://github.com/anvaka/map-of-github

https://anvaka.github.io/map-of-github/ - here it is

I'm quite unhappy with SBCL located in Schemaria:

https://anvaka.github.io/map-of-github/#12/13.469/-8.175

It should be Lispaña.

I had one of these recently! https://github.com/simonw/shot-scraper/pull/133/files

They're /incredibly/ rare though.

If anyone wants the raw data, it's available in window._Flourish_data variable on https://flo.uri.sh/visualisation/16818696/embed

Which means you can extract it with my https://shot-scraper.datasette.io/ tool like this:

    shot-scraper javascript \

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with.

I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable/javascript.html

This is exactly what the `Accept` HTTP header is for https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ac...

I think the author is generally correct that all JSON should be provided in a single request, but if you want to prove it, then you should be able to change your accept header to and from `application/json`/`text/html seeing nearly identical data.

In fact, this is what both GitLab and Github do. Try it out!

`curl -L https://github.com/simonw/shot-scraper` (text/html)

`curl --header "Accept: application/json" -L https://github.com/simonw/shot-scraper` (application/json)

Git is a key technology in this approach, because the value you get out of this form of scraping is the commit history - it's a way of turning a static source of information into a record of how that information changed over time.

I think it's fine to use the term "scraping" to refer to downloading a JSON file.

These days an increasing number of websites work by serving up JSON which is then turned into HTML by a client-side JavaScript app. The JSON often isn't a formally documented API, but you can grab it directly to avoid the extra step of processing the HTML.

I do run Git scrapers that process HTML as well. A couple of examples:

scrape-san-mateo-fire-dispatch https://github.com/simonw/scrape-san-mateo-fire-dispatch scrapes the HTML from http://www.firedispatch.com/iPhoneActiveIncident.asp?Agency=... and records both the original HTML and converted JSON in the repository.

scrape-hacker-news-by-domain https://github.com/simonw/scrape-hacker-news-by-domain uses my https://shot-scraper.datasette.io/ browser automation tool to convert an HTML page on Hacker News into JSON and save that to the repo. I wrote more about how that works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

You can use shot-scraper: https://github.com/simonw/shot-scraper

I have a neat Hacker News scraping setup that I'm really pleased with.

The problem: I want to know when content from one of my sites is submitted to Hacker News, and keep track of the points and comments over time. I also want to be alerted when it happens.

Solution: https://github.com/simonw/scrape-hacker-news-by-domain/

This repo does a LOT of things.

It's an implementation of my Git scraping pattern - https://simonwillison.net/2020/Oct/9/git-scraping/ - in that it runs a script once an hour to check for more content.

It scrapes https://news.ycombinator.com/from?site=simonwillison.net (scraping the HTML because this particular feature isn't supported by the Hacker News API) using shot-scraper - a tool I built for command-line browser automation: https://shot-scraper.datasette.io/

The scraper works by running this JavaScript against the page and recording the resulting JSON to the Git repository: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

That solves the "monitor and record any changes" bit.

But... I want alerts when my content shows up.

I solve that using three more tools I built: https://datasette.io/ and https://datasette.io/plugins/datasette-atom and https://datasette.cloud/

This script here runs to push the latest scraped JSON to my SQLite database hosted using my in-development SaaS platform, Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/blob/...

I defined this SQL view https://simon.datasette.cloud/data/hacker_news_posts_atom which shows the latest data in the format required by the datasette-atom plugin.

Which means I can subscribe to the resulting Atom feed (add .atom to that URL) in NetNewsWire and get alerted when my content shows up on Hacker News!

I wrote a bit more about how this all works here: https://simonwillison.net/2022/Dec/2/datasette-write-api/

Sounds a lot like Simon Willison's open source project shot-scraper

https://github.com/simonw/shot-scraper