warcprox VS webcrystal

I've looked into archiving all the pages i visit as well and warcprox[1] has been bookmarked for a while now

Hard drive storage space being so cheap in the ~$15/TB range makes this more feasible even for video archival

[1] https://github.com/internetarchive/warcprox

wget also supports WARC options if you don't need javascript etc. If you do, there's also Warcprox (https://github.com/internetarchive/warcprox), brozzler (https://github.com/internetarchive/brozzler) (which uses warcprox internally), and others.

Wow, it's really rare these days to see a tool that supports WARC.

Despite being an ISO standard [1] and the default archive format of the internet archive, and despite a handfull of lovingly crafted tools (such as webrecorder [2], warcprox etc.), it never seems to have caught on in a broader context.

Really a shame - I' deeply convinced that the ability to archive and replay requests is a technique for defending and strengthening user rights.

Links:

[1] https://www.iso.org/standard/44717.html

[2] https://github.com/webrecorder/webrecorder-desktop

[3] https://github.com/internetarchive/warcprox

However, if you prefer something open-source, you could accomplish the same thing with a tool like https://archiveweb.page/ or https://github.com/internetarchive/warcprox

While it lacks a search feature last I checked there's always https://github.com/davidfstr/webcrystal

One .py file. Only one dependency (urllib3).n with a little love the concept could become a full transparent proxy.

From the the project page it says:

> The offline content is stored in ~/.cache/offpunk/ as plain .gmi/.html files. The structure of the Gemini-space is tentatively recreated. One key element of the design is to avoid any database. The cache can thus be modified by hand, content can be removed, used or added by software other than offpunk.

One ambition I have it to setup

https://github.com/davidfstr/webcrystal

> An archiving HTTP proxy and on-disk archival format for websites.

so that all my regular web browsing is auto archived at some level.

It would sure be neat if the archive formats could be compatible. It would allow for a setup where everything I’ve seen with my eyes is then immediately accessible programmatically or in a terminal. I feel that could open some significant productive advantages, especially in the age of LLMs also in the terminal.

Webcrystal?