pypandoc VS panflute

I recently used [0] Playwright for Python and [1] pypandoc to build a scraper that fetches a webpage and turns the content into sane markdown so that it can be passed into an AI coding chat [2].

They are both very gentle dependencies to add to a project. Both packages contain built in or scriptable methods to install their underlying platform-specific binary dependencies. This means you don't need to ask end users to use some complex, platform-specific package manager to install playwright and pandoc.

Playwright let's you scrape pages that rely on js. Pandoc is great at turning HTML into sensible markdown. Below is an excerpt of the openai pricing docs [3] that have been scraped to markdown [4] in this manner.

[0] https://playwright.dev/python/docs/intro

[1] https://github.com/JessicaTegner/pypandoc

[2] https://github.com/paul-gauthier/aider

[3] https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...

[4] https://gist.githubusercontent.com/paul-gauthier/95a1434a28d...

  ## GPT-4 and GPT-4 Turbo

Use Pypandoc

Interesting idea re:internal links. For sufficiently complex issues of this nature, pandoc filters[0] are a powerful tool for this kind of mid-conversion processing. I've made some cool projects with the Python package panflute[1]

[0] https://pandoc.org/filters.html

[1] https://github.com/sergiocorreia/panflute

Something that is quite suited to little mini projects with python is writing pandoc filters. The basic idea of pandoc is to convert written documents of some file type (the main source type is typically markdown) into another document type, for example latex, pdf, docx, odt... Now of course there will be certain things that pandoc cannot do quite well enough given its versatility. So it gives users the option to write [simple filters](https://pandoc.org/filters.html#summary) and apply them. This way you can also come up with your own syntax/tags and whatever in markdown and convert that to an arbitrary formatting instruction in another document type. Look into something like panflute and maybe you can come up with your own filters.