undetected-chromedriver VS estela

This command-line tool clicks ads for a certain query on Google/Bing search using undetected_chromedriver package. Supports proxy, running multiple simultaneous browsers, ad targeting/exclusion, and running in loop.

You can check this out. https://github.com/ultrafunkamsterdam/undetected-chromedriver If i didn't understand you wrong then this is what you're asking for.

403 often means that the server recognized the scraper and blocked you. If you use Selenium, this plugin is very good for passing bot detection: https://github.com/ultrafunkamsterdam/undetected-chromedriver.

Ultrafunkamsterdam created a ChromeDriver that has excellent support for bypassing all major bot detection systems such as Distil, Datadome, Cloudflare, and others.

One solution would be to install Selenium and then scrape using a real browser like Chrome. If this solution gets blocked, you could install obfuscation plugins like this very good one: https://github.com/ultrafunkamsterdam/undetected-chromedriver

Undetected_ChromeDriver also works on Brave Browser and many other Chromium-based browsers. For more, you can check out this project on GitHub.

This solution is using requests, you can also do this in scrapy, and if you are planning to run more crawlers you can use estela which is a spider management solution.

You may want to check out [estela](https://estela.bitmaker.la/docs/), which is a spider management solution, developed by [Bitmaker](https://bitmaker.la) that allows you to run [Scrapy](https://scrapy.org) spiders.

We are currently running a closed beta of Bitmaker Cloud (free and unlimited). Bitmaker Cloud gives you easy management of scraping workloads via a web dashboard and API. Only Scrapy spiders are supported at the moment (additional languages/frameworks are on the roadmap). Bitmaker Cloud is powered by estela, an elastic web scraping cluster running on Kubernetes. estela is a modern alternative to proprietary platforms such as Scrapy Cloud, as well as OSS projects such as scrapyd. The source code of estela and estela-cli is available on Github.

Estela: A webscraping framework on to of Kubernetes, which manage scaling (by Breno Colom)

We released estela for this and other purposes, check it out, maybe it will suit your needs:

https://github.com/bitmakerla/estela

Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. Would be good to know which ones to prioritize over others :-)