github-explorer VS map-of-github

clickhouse has pretty good github_events dataset on their playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

It shouldn’t be hard to implement: https://ghe.clickhouse.tech/#how-to-download-the-data

The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

See also https://til.simonwillison.net/clickhouse/github-explorer

I recommend checking https://ghe.clickhouse.tech/

It explains the full pipeline - how to download, collect, and analyze this sort of data.

Yes, it's continuously updated.

The source code is here: https://github.com/ClickHouse/github-explorer

This shell scripts updates it: https://github.com/ClickHouse/github-explorer/blob/main/upda...

You may appreciate the map of github [0] it's a fantastic piece of work that trended here a while back.

[0]: https://anvaka.github.io/map-of-github/

https://www.reddit.com/r/dataisbeautiful/comments/i8saks/ive...

https://www.reddit.com/r/dataisbeautiful/comments/ic2i0k/oc_...

Stock chart as a landscape image

And I also like these "races" basically a timelapse of a chart

https://www.reddit.com/r/dataisbeautiful/comments/f68fzr/oc_...

And whatever anvaka does, all can be tried out in the browser

https://github.com/anvaka?tab=repositories&q=&type=&language...

e.g. map of reddit:

https://github.com/anvaka/map-of-reddit

Video Demo: https://www.reddit.com/r/dataisbeautiful/comments/12pem68/oc...

Similarly map of github

https://github.com/anvaka/map-of-github

Or package manager visualizations:

https://github.com/anvaka/pm

Google search completion visualizations

https://github.com/anvaka/vs

I don't find these argument convincing

- hackers in Shenzhen are underrepresented b/c of the language barrier. There is actually a huge Chinese language ecosystem (a lot of it off Github on Gitee) that is sort of impenetrable but you come across it all the time. I just typically have no way to use their work b/c it's under-documented in English. If you zip around this map you will see lots of "Zh" islands: https://anvaka.github.io/map-of-github/

- While I understand that it's not as financial comfortable for a program in Indian compared wtih the US or Europe, Chinese and Russian hackers aren't significantly better off. Your typical Chinese program is also of course worried about working for a big company and making money and all of that jazz

- "escape the hell that is 'unpaid open source work' " - I think maybe is the crux of it... Most people work on open source work b/c they enjoy it.. I don't personally know anyone who got hired from a deliberate effort to build an open source portfolio. I wouldn't recommend it to anyone as a way to try to find a job

The source code with description of the method is available here: https://github.com/anvaka/map-of-github

https://anvaka.github.io/map-of-github/ - here it is

I'm quite unhappy with SBCL located in Schemaria:

https://anvaka.github.io/map-of-github/#12/13.469/-8.175

It should be Lispaña.