wayback VS cloc

In this tutorial, we will explore how to scrape data from the past using the Wayback Machine API. We'll be using Python and the requests library to make HTTP requests and retrieve archived versions of web pages. The code provided demonstrates a basic implementation of scraping historical data from a list of URLs within a specified date range.

A bit of context: I do a lot of archive digging with Apple's website. For the longest time, they hosted large files through Akamai. Most URLs looked something like this: http://a2032.g.akamai.net/5/2032/51/6cafb32dc21f74/1a1a1aaa2198c627970773d80669d84574a8d80d3cb12453c02589f25382f26493036bda4ebd305fd241a71b92f365ca/appleworks62_box.eps.hqx Unfortunately, those files shifted around from subdomain to subdomain (one period of time it was under a2032.g.akamai.net, another might be a1008.g.akamai.net) so finding all copies of a specific file was a pain in the ass. I recently learned that the IA has an API for the Wayback's Server that allows way more filtering than the web UI does. So to find every *.g.akamai.net URL they have archived, I used: http://web.archive.org/cdx/search/cdx?url=*.g.akamai.net/*

For the following step, you will need the Wayback Machine's CDX API, the documentation is here:https://github.com/internetarchive/wayback/tree/master/wayback-cdx-serverHowever, please note that there are errors in the documentation regarding the regex filtering syntax.

archive.org geocities scrapes go back to 1996, so it is plausible it could have survived:

https://web.archive.org/cdx/search/cdx?url=geocities.com&mat...

If you ever remember any of the details, the CDX API can probably help.

https://github.com/internetarchive/wayback/blob/master/wayba...

Query the Internet Archive’s CDX server for this info.

https://github.com/internetarchive/wayback/blob/master/wayba...

Archive.org has a cdx server you can quickly request information from: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

Are the contents of these WARCs available as part of the Wayback Machine itself? If so, you might be able to use the CDX server to discover and download the content.

Wayback API to get a list of all versions of the page (https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server#basic-usage).

Cloc - https://github.com/AlDanial/cloc

and get the results on the cli.

Let me know what you think. :)

0: https://github.com/AlDanial/cloc

Awesome stuff, thank you! I‘d love some flags/options for cloc integration if it can be detected, maybe a summary of the top N languages for directories (67% Rust, 13% Html, 9% Bash) or something. Just a suggestion/idea. Gonna install it anyway, it‘s shiny!

The solution I am using is currently comprised of 145 projects, 141k+ lines of C#, and 37k+ lines of Razor, courtesy of cloc:

Just out of curiosity, I downloaded the latest version of GNU coreutils and compared the line count between a few source files and the resulting disassembled object files (using cloc to exclude blank lines and comments). It looks like the ratio is very approximately 2 assembly instructions per line of C code. Obviously, that will depend a lot on what the code is doing and the coding style.

When I started making a game [0] last year, first thing I did was write a little Unity script that takes a screenshot of the opening scene, counts current lines of code using CLOC [1] (for fun, not as a true measure of anything), and occasionally renders it all out to an image file.

With that I'm able to create some pretty fun time lapses of progress. I've been doing this at an arbitrary milestone, whenever my Luau [2] LOC surpasses C++ by another factor. This post reminded me I'm overdue for another now that Luau > 3x C++ LOC.

I find it rewarding to look back at my progress. I'll share in case it's interesting for you too [3].

[0] https://store.steampowered.com/app/2168330/Helmscape/

[1] https://github.com/AlDanial/cloc

[2] https://luau-lang.org

[3] https://twitter.com/kineticpoet/status/1619508466212831232

My manager just asked me about this a few days ago (sigh) cloc is good for this - you can pass it a hash or two hashes and it will give you counts accordingly. https://github.com/AlDanial/cloc

Finally, since I'm limited on the character-length of this post, I'll post an individual comment for each year with a table of data. The "All Rank" column will rank the problem by difficulty (measured by leaderboard close time) across all years, with 1 being longest. The "Yr Rank" column will be similar, but ranked only within that year. The "P1 LOC" and "P2 LOC" columns show the numbers of lines of code in my solutions for each part as measured by cloc (each part is stand-alone so there will be some duplication, especially for Intcode). Other columns should be self-explanatory.