rymscraper VS dude

True. I use a scraper that someone built (https://github.com/dbeley/rymscraper) + rate limiting, it seems to work just fine without getting my IP banned, although it is necessarily slow. I would just run it overnight or so.

Please check https://github.com/roniemartinez/dude

You can use my library which also uses Playwright. I have an example here: https://github.com/roniemartinez/dude/discussions/116

I also built a framework so I can easily switch between these libraries with less code change (still on hiatus for a few months before going back to it): https://github.com/roniemartinez/dude

There's a lot of options but I am quite happy with Github Actions workflows + Poetry as it handles tests and publish to PyPI. Just an example, in my workflows, I deploy to TestPyPI and PyPI here: https://github.com/roniemartinez/dude/tree/master/.github/workflows

But for documentation, I use mkdocs-material as it can easily be used with minor customization and changes can be easily deployed in Github: https://roniemartinez.github.io/dude/

Add an adblocker. I implemented Dude/pydude with the this and page results are clean without ads and pop-ups. For the screenshot, here is an example: https://github.com/roniemartinez/dude/discussions/116

You can also use my library if you want things to be simpler:) https://github.com/roniemartinez/dude

I use my own library, Dude! https://github.com/roniemartinez/dude

I built a simple web scraper that is simple to use but this is still a work-in-progress - https://github.com/roniemartinez/dude