Pack VS github-explorer

The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)

It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).

I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.

[0] https://pack.ac/

[1] https://news.ycombinator.com/item?id=39793805

[2] https://github.com/PackOrganization/Pack/issues/10

clickhouse has pretty good github_events dataset on their playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/

Example of what user JiaT75 did so far:

https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days:

https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

It shouldn’t be hard to implement: https://ghe.clickhouse.tech/#how-to-download-the-data

The article leaves a bitter taste of unnecessary complexity. Data engineering should not be hard.

For example, you can load the GitHub Archive to ClickHouse, and it will be accessible with interactive real-time queries: https://ghe.clickhouse.tech/

See also https://til.simonwillison.net/clickhouse/github-explorer

I recommend checking https://ghe.clickhouse.tech/

It explains the full pipeline - how to download, collect, and analyze this sort of data.

Yes, it's continuously updated.

The source code is here: https://github.com/ClickHouse/github-explorer

This shell scripts updates it: https://github.com/ClickHouse/github-explorer/blob/main/upda...