scrapyd VS linkedom

Scrapyd is a good option to run your scrapers remotely in the cloud. Adding a Scrapyd dashboard makes the experience better.

8. If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd.

Scrapyd is one of the most popular options. Created by the same developers that developed Scrapy itself, Scrapyd is a tool for running Scrapy spiders in production on remote servers so you don't need to run them on a local machine.

ScrapydWeb is the most popular open source Scrapyd admin dashboards. Boasting 2,400 Github stars, ScrapydWeb has been fully embraced by the Scrapy community.

or scrapyd -> https://github.com/scrapy/scrapyd

While I am aware that we could use Scrapyd to host your spiders and actually send requests, alongside with ScrapydWeb, I personally prefer to keep my scraper deployment simple, quick, and free. If you are interested in this alternative instead, check out this post written by Harry Wang.

linkedom is your answer

linkedom - to parse HTML into a workable DOM. I used to use jsdom, but I switch for performance reasons.

For simple scraping where the content is fairly static, or when performance is critical, I will use linkedom to process pages.

https://github.com/WebReflection/linkedom

When the content is complex or involves clicking, Playwright is probably the best tool for the job.

https://github.com/microsoft/playwright

I recently started using linkedom for this and it has been an absolute joy

This looks great. I wonder how it compares to linkedom (repo[1], writeup[2]), which I have found to be fantastic.

[1]: https://github.com/WebReflection/linkedom

[2]: https://webreflection.medium.com/linkedom-a-jsdom-alternativ...

linkedom, fastest, but lacks essential features

Other things to do would be to use linkedom instead of JSDom, look into a better assertion library than assert etc.

Sounds somewhat similar to linkedom[1], which performs nicely.

[1]: https://github.com/WebReflection/linkedom