parser VS pandoc

Thoroughly scraping is challenging, especially in an environment where you don’t have (or want) a JavaScript runtime.

For content extraction, I found the approach the Postlight library takes quite neat. It scores individual html nodes based on some heuristics (text length, link density, css classes). It the selects the nodes with the highest score. [1] I ported it to Swift for a personal read later app.

[1] https://github.com/postlight/parser

I've been working on an enhanced reader mode extension for the last few months. I found that Mercury Reader's parser tool is useful for extracting content. If that's not exactly what you're looking for, readibility is another good option. It's a library used inside Firefox's reader moder that you can use in any project.

I currently have it turned off while I search for better sources, but I have a VM that runs a custom cron script that combines a custom RSS reader, podfox, mercury-parser, and coqui-ai to generate audio podcasts from RSS news feeds. I should probably clean it up and release the script/setup process. With a few tweaks and some AI text-to-speech and a little machine learning audio processing you can get a really good podcast experience from text posts.

It looks like Relay could be updated to convert it locally though, since the parser that it uses appears to be open source.

https://github.com/postlight/mercury-parser The only one I need, shit's too good

An alternative would be to extract the plain text from the article's page with either some "readability" API or a library like Mercury Parser: https://github.com/postlight/mercury-parser

I haven’t directly compared them, but I have also found mercury parser (https://github.com/postlight/mercury-parser) to be very reliable.

Since it turns a website into very plain (X)HTML it‘s fairly easy to use it to make a browsing proxy or automatically produce epub files for e-readers, which is what I do.

Make sure that at the end of the process you have the node and npm executables installed - the http.webpage integration uses the Mercury Parser API to convert web pages to Markdown.

Good luck! Those HTML issues you're coming across are tough and so varied across the web!

I was working with Mercury Parser (pluggable parsing for different sites) in the past.

https://github.com/postlight/mercury-parser

My main authoring tool is then Emacs Markdown Mode (https://jblevins.org/projects/markdown-mode/). For data entry, it comes with some bells and whistles similar to org-mode, like C-c C-l for inserting links etc.

I seldom export my notes for external usage, but if it is the case, I use lowdown (https://kristaps.bsd.lv/lowdown/) which also comes with some nice output targets (among the more unusual are Groff and Terminal). Of cource pandoc (https://pandoc.org/) does a very good job here, too.

This is one of those things that the ever-amazing pandoc (https://pandoc.org/) does very well, on top of supporting virtually every other document format.

Folks feel the same way about Markdown versus LaTeX: why use something significantly more complicated where a looser, human-readable grammar works better?

For any other situations, I use https://pandoc.org/, or, generate a Word doc scriptomatically.

pandoc toolchain pour builder une version confortable/imprimable en phase de travail (ePub, pdf, docx, html)

Congrats on the launch, I guess, but there are so many free options that I can't think of a situation where paying $0.25 per document would be justified...? Just to name a few:

Back in the days, I used to use XSL-FO [0] and it was okay. It was not very precise but it rarely if ever broke, and was perfectly integrated with an XML/XSLT solution. Yeah, this was a long time ago.

Last month I used html-to-pdfmake [1] and it's also not very precise and more fragile, but very efficient and fast.

Yet another approach would be to pro grammatically generate .rtf files (for example) and use Pandoc [2] to produce PDFs (I have not tried this in production but don't see why it wouldn't work).

[0] https://en.wikipedia.org/wiki/XSL_Formatting_Objects

[1] https://www.npmjs.com/package/html-to-pdfmake

[2] https://pandoc.org/

Others have mentioned static site generators. I like Hakyll [1] because it can tightly integrate with Pandoc [2] and allows you to develop custom solutions if your needs ever grow.

[1]: https://jaspervdj.be/hakyll/

[2]: https://pandoc.org/

Have you compared it with a conversion by pandoc (https://pandoc.org/)?

I have used it to kickstart a blogging project that I wish to come back to soon. The Lua inter-op for custom readers, writers and filters is great but I wish there was more editor integration and even perhaps an official IDE/editor with built-in debugging features (probably something already do-able with Emacs but I haven't checked). The only blocker for my project is no support for "ChunkedDoc" for Lua filters [1] which forces me to write more code and a complicated Makefile.

[1]: https://github.com/jgm/pandoc/issues/9061