go-readability VS wayback-machine-downloader

I've had a lot of success by running HTML pages through mozilla's readability[0] tool (actually the go port of it[1]) before indexing it.

[0]: https://github.com/mozilla/readability

[1]: https://github.com/go-shiori/go-readability

https://github.com/go-shiori/go-readability https://github.com/mauidude/go-readability

I'm using https://github.com/go-shiori/go-readability -- a Go re-implementation of Mozilla's readability-js library. It does a pretty good job.

Two projects that do this with nearly identical output:

- https://github.com/eafer/rdrview

- https://github.com/go-shiori/go-readability

Pipe the filtered HTML output into your favorite textual web browser for an ideal reading experience.

I just found this https://github.com/hartator/wayback-machine-downloader

anyone have anything similarly interesting/cool/niche-useful ?

Send ArchiveTeam the link on IRC or here and we can save it to archive.org, then later you can use wayback-machine-downloader to grab it from archive.org.

https://github.com/hartator/wayback-machine-downloader

This is where it gets tricky, you need to download the code from the wayback machine and he was able to do that by following these steps: https://github.com/hartator/wayback-machine-downloader

Not sure if it will work for twitter, but I have used wayback-machine-downloader to batch download stuff.

Adding to archive.org, there is a github repo to fetch website data. You can give a try too. Here is the repo link

Edit: Good news! I found a solution that saved me. I was able to download the full website (including images, JS, and CSS files) using this tool: https://github.com/hartator/wayback-machine-downloader

Edit: Thank you to everyone who suggested the Wayback Machine Downloader! It saved the day and allowed me to download the full website, including images, JS, and CSS files.

I've used https://github.com/hartator/wayback-machine-downloader but it sometimes messes up CSS badly