BasicCrawler VS rdrview

Very cool.

The take-any-webpage-offline need is also common in the education space (teachers want to save a webpage and send it to their students as part of a lesson and don't want to worry about availability or ads etc).

I used to work on tools for this https://github.com/learningequality/ricecooker/blob/develop/... and https://github.com/learningequality/BasicCrawler/blob/master...

See also the C port here: https://github.com/eafer/rdrview/

It works well with text-mode browsers like w3m.

i also use rdrview sometimes.

They both parse untrusted content content without sandboxing.

I typically send content through rdrview[0] before piping through w3m-sandbox[1], which should be pretty safe.

[0]: https://github.com/eafer/rdrview

[1]: https://git.sr.ht/~seirdy/bwrap-scripts/tree/trunk/item/w3m-...

Could have been nice to have this integrated to w3m. Somthing along the lines of rdrview.

I do a lot of this work[3] (web to documents) and it's interesting to see other approaches. The medium image problem is something I've faced as well, but never got around to fixing. I'm planning to get a Remarkable soon, so will definitely be trying this out.

My personal solution has been https://github.com/captn3m0/url-to-epub/ (Node/readability), which I've tested against the entirety of Tor's original fiction collection[0] where it performs well enough (I'm biased). Another tool that does this beautifully well is percollate[1], but it doesn't give enough control of the metadata to the user - something I really care about.

I've also started to use rdrview[2], which is a C-port of the current Firefox implementation of "reader view". It is very unix-y, so it is easy to pipe content to it (I usually run it through tidy first). Quite helpful in building web-archiving or web-to-pdf or web-to-kindle pipelines easily.

[0]: https://www.tor.com/category/all-fiction/original-fiction/

[1]: https://github.com/danburzo/percollate

[2]: https://github.com/eafer/rdrview

[3]: https://captnemo.in/ebooks/

Two projects that do this with nearly identical output:

- https://github.com/eafer/rdrview

- https://github.com/go-shiori/go-readability

Pipe the filtered HTML output into your favorite textual web browser for an ideal reading experience.

This may help if you can do some piping around it.. https://github.com/eafer/rdrview

SEEKING WORK | Argentina | Remote

Email: [email protected]

I'm a programmer, most familiar with C on Linux and Win32. I'll be happy to start a project from scratch, or to help support any old codebase. For a sample of my work please see rdrview [1], a small command line tool that found some success here on Hacker News; or [2], a naive filesystem implementation I've been working on.

My current rate is 20 USD/hour. For what it's worth, I have a background in math.

[1] https://github.com/eafer/rdrview

[2] https://github.com/linux-apfs