github-explorer VS demo

clickhouse has pretty good github_events dataset on their playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

It shouldn’t be hard to implement: https://ghe.clickhouse.tech/#how-to-download-the-data

The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

See also https://til.simonwillison.net/clickhouse/github-explorer

I recommend checking https://ghe.clickhouse.tech/

It explains the full pipeline - how to download, collect, and analyze this sort of data.

Yes, it's continuously updated.

The source code is here: https://github.com/ClickHouse/github-explorer

This shell scripts updates it: https://github.com/ClickHouse/github-explorer/blob/main/upda...

The "This is fine" achievement is funny but doesn't seem right. The top user ppdemo indeed has a repo where he apparently creates one issue every minutes, but there's only a few open issues so he must be deleting them too:

https://github.com/pddemo/demo

So I recently found about github.com/pddemo/demo which has a bot create lots of issues, around 600k issues, and the issue numbers are 59X,XXX but I'm trying to find a Github repo with more issues or higher issue numbers, answers telling me to find a repo with the most issues/highest issue numbers using API are also welcome.