linkedom VS estela

linkedom is your answer

linkedom - to parse HTML into a workable DOM. I used to use jsdom, but I switch for performance reasons.

For simple scraping where the content is fairly static, or when performance is critical, I will use linkedom to process pages.

https://github.com/WebReflection/linkedom

When the content is complex or involves clicking, Playwright is probably the best tool for the job.

https://github.com/microsoft/playwright

I recently started using linkedom for this and it has been an absolute joy

This looks great. I wonder how it compares to linkedom (repo[1], writeup[2]), which I have found to be fantastic.

[1]: https://github.com/WebReflection/linkedom

[2]: https://webreflection.medium.com/linkedom-a-jsdom-alternativ...

linkedom, fastest, but lacks essential features

Other things to do would be to use linkedom instead of JSDom, look into a better assertion library than assert etc.

Sounds somewhat similar to linkedom[1], which performs nicely.

[1]: https://github.com/WebReflection/linkedom

This solution is using requests, you can also do this in scrapy, and if you are planning to run more crawlers you can use estela which is a spider management solution.

You may want to check out [estela](https://estela.bitmaker.la/docs/), which is a spider management solution, developed by [Bitmaker](https://bitmaker.la) that allows you to run [Scrapy](https://scrapy.org) spiders.

We are currently running a closed beta of Bitmaker Cloud (free and unlimited). Bitmaker Cloud gives you easy management of scraping workloads via a web dashboard and API. Only Scrapy spiders are supported at the moment (additional languages/frameworks are on the roadmap). Bitmaker Cloud is powered by estela, an elastic web scraping cluster running on Kubernetes. estela is a modern alternative to proprietary platforms such as Scrapy Cloud, as well as OSS projects such as scrapyd. The source code of estela and estela-cli is available on Github.

Estela: A webscraping framework on to of Kubernetes, which manage scaling (by Breno Colom)

We released estela for this and other purposes, check it out, maybe it will suit your needs:

https://github.com/bitmakerla/estela

Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. Would be good to know which ones to prioritize over others :-)