warcprox VS TWINT

I've looked into archiving all the pages i visit as well and warcprox[1] has been bookmarked for a while now

Hard drive storage space being so cheap in the ~$15/TB range makes this more feasible even for video archival

[1] https://github.com/internetarchive/warcprox

wget also supports WARC options if you don't need javascript etc. If you do, there's also Warcprox (https://github.com/internetarchive/warcprox), brozzler (https://github.com/internetarchive/brozzler) (which uses warcprox internally), and others.

Wow, it's really rare these days to see a tool that supports WARC.

Despite being an ISO standard [1] and the default archive format of the internet archive, and despite a handfull of lovingly crafted tools (such as webrecorder [2], warcprox etc.), it never seems to have caught on in a broader context.

Really a shame - I' deeply convinced that the ability to archive and replay requests is a technique for defending and strengthening user rights.

Links:

[1] https://www.iso.org/standard/44717.html

[2] https://github.com/webrecorder/webrecorder-desktop

[3] https://github.com/internetarchive/warcprox

However, if you prefer something open-source, you could accomplish the same thing with a tool like https://archiveweb.page/ or https://github.com/internetarchive/warcprox

twint is a project that can scrape twitter data via the webpages rather than the twitter API, which means that it can get more than the last 3200 tweets of an account. Unfortunately it seems that the repo was archived and is no longer in development, so I'm not sure if it even still works. It's also a bit heavy on dependencies and is written in Python, neither of which make it easier to install and use.

The tool doesn't work anymore since Twitter changed its APIs, but a good example is twint. Most people in OSINT are not highly technical and don't know their way around a CLI. On the other hand, a CLI tool is one of the quickest, lowest (dev) cost ways to release a tool to the public, and many developers who build tools for the OSINT community do so for free (open source).

You can try twint which is extensive and should be able to do that. Another is via this twitter downloader but might require multiple runs depending on what you want to archive.

They do have examples that don't specify a username such as number 3 on this page or this one on the main page: "twint -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file."

This will motivate researchers to web scrape to circumvent these restrictions. Twint can scrape tweets and it supports proxies. It can also be multi threaded. A huge hassle and it's prone to breaking when the site changes.

The comment I saw said they used this: https://github.com/twintproject/twint