VaRA-Tool-Suite VS github-explorer

I tried to understand the significance of this (parent maybe implied that they reused a completely fictitious identity generated by some test code), and I think this is benign.

That project just includes some metadata about a bunch of sample projects, and it links directly to a mirror of the xz project itself:

https://github.com/se-sic/VaRA-Tool-Suite/blob/982bf9b9cbf64...

I assume it downloads the project, examines the git history, and the test then ensures that the correct author name and email addresses are recognized.

(that said, I haven't checked the rest of the project, so I don't know if the code from xz is then subsequently built, and or if this other project could use that in an unsafe manner)

clickhouse has pretty good github_events dataset on their playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

It shouldn’t be hard to implement: https://ghe.clickhouse.tech/#how-to-download-the-data

The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

See also https://til.simonwillison.net/clickhouse/github-explorer

I recommend checking https://ghe.clickhouse.tech/

It explains the full pipeline - how to download, collect, and analyze this sort of data.

Yes, it's continuously updated.

The source code is here: https://github.com/ClickHouse/github-explorer

This shell scripts updates it: https://github.com/ClickHouse/github-explorer/blob/main/upda...