lxml VS html5lib

lxml bumped from version 4.9.3 to 5.3.0, see release notes

My journey in web scraping started in 2016 when I was solving lead generation challenges for a small company. Initially, I used off-the-shelf solutions such as Import.io and Kimono Labs. However, I quickly encountered limitations such as blocking, inaccurate data extraction, and performance issues. This led me to learn Python. Those were the glory days when requests and lxml/beautifulsoup were enough to extract data from most websites. And if you knew how to work with threads, you were already a respected expert :)

lxml

Parsel is a library build to extract data from XML/HTML documents with XPath and CSS selectors support, and could be combined with regular expressions. It's usees lxml parser under the hood by default.

So, now that I no longer work in that industry, and I had some free time, I created a lazy parsing package using lxml instead of the xml parser in the standard library, which can help people who want to have a python only parsing solution.

Seriously, use LXML.

I got curious about this now so I did some tests on my own, and it appears that the XML parser implementation in Python does indeed strip all newline characters from attributes. Whether this is according to XML standard I do not know; I also briefly tried an alternative XML implementation for Python and it behaves the same, so I would assume that this is standard behavior, but I'm not knowledgable enough about XML to say for certain.

html5lib-python – Standards-compliant library for parsing and serializing HTML documents and fragments in Python

Yes. This is really interesting.

Sounds like html5lib has been asking for funding, but doesn't look like there's much progress. https://github.com/html5lib/html5lib-python/issues/361

Neither did html5lib.

Easier example...sphinx is a document generator for python programs (creating docs for the API of programs from source-code comments for example). Spinx depends on html5lib which itself again depends on six...want to make a guess what six uses to generate its API docs? ;) So if you want the api docs of six you will have to first install it without to be able to get a working sphinx install then redo the six on including the building of the API docs.