go-readability VS wayback-machine-downloader

https://github.com/go-shiori/go-readability https://github.com/mauidude/go-readability

In terms of extracting the actual blog content from pages, there is a go library that implements the readability algorithm:

https://github.com/mauidude/go-readability

This is the kind of thing pocket/instapaper do to extract the main content from a page in a format that's easier to read (and also probably to programmatically modify)

I just found this https://github.com/hartator/wayback-machine-downloader

anyone have anything similarly interesting/cool/niche-useful ?

Send ArchiveTeam the link on IRC or here and we can save it to archive.org, then later you can use wayback-machine-downloader to grab it from archive.org.

https://github.com/hartator/wayback-machine-downloader

This is where it gets tricky, you need to download the code from the wayback machine and he was able to do that by following these steps: https://github.com/hartator/wayback-machine-downloader

Not sure if it will work for twitter, but I have used wayback-machine-downloader to batch download stuff.

Adding to archive.org, there is a github repo to fetch website data. You can give a try too. Here is the repo link

Edit: Good news! I found a solution that saved me. I was able to download the full website (including images, JS, and CSS files) using this tool: https://github.com/hartator/wayback-machine-downloader

Edit: Thank you to everyone who suggested the Wayback Machine Downloader! It saved the day and allowed me to download the full website, including images, JS, and CSS files.

I've used https://github.com/hartator/wayback-machine-downloader but it sometimes messes up CSS badly