undetected-chromedriver VS puppeteer-extra

This command-line tool clicks ads for a certain query on Google/Bing search using undetected_chromedriver package. Supports proxy, running multiple simultaneous browsers, ad targeting/exclusion, and running in loop.

Ultrafunkamsterdam created a ChromeDriver that has excellent support for bypassing all major bot detection systems such as Distil, Datadome, Cloudflare, and others.

Undetected_ChromeDriver also works on Brave Browser and many other Chromium-based browsers. For more, you can check out this project on GitHub.

Thanks a lot for your help. You were very passionate about it which means you cared. My mom called all my aunts and cousins gushing that I have written a project in 6 languages, I mostly look for people to fix the mistakes I have made even if the entire thing is a mistake. I think you and another person on C subreddit are the only person who went through this code. The person on that sub marked a mistake I have made regarding C and you marked the problem with everything which means I have gotten enough out of this. I've seen stupid people release bullshit "libraries" and stupider people using them and never know that they are in the wrong because they take every criticism as an insult. I told this genius that not only his conduct is extremely unprofessional but that he's misleading everyone because the usernames people use to access those sites will eventually get banned and he will have thousands of dollars people paid for botted users on his conscious. But he cringefully ignored me. He's such a dummy he's added a freaking shebang to files that are meant to be ran as a library! Jesus. Sorry I'm meandering.

There is a good chance that the website is using Cloudflare to block web scrapers, which will require you to use a fortified headless browser to solve the JS challenges. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

I have used this project before. I think the author probably can create something lightweight with his knowledge. For my purposes I couldn’t find a Cloudflare bypass script that worked so I had to use this instead.

https://github.com/ultrafunkamsterdam/undetected-chromedrive...

Modded chromedriver + rotating user string + some reliable proxy should knock simple systems. Some anti-scraping systems are quite advanced, however.

Fortified Headless Browser: Depending on the anti-bot protection the website is using you make need to use a fortified headless browser that can solve its JS challenges without giving its identity away. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

There is a good chance that the website is using Cloudflare to block web scrapers, which will require you to use a fortified headless browser to solve the JS challenges. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

There are even Puppeteer plugins that will do it for you. [^1]

The best detection I've come across so far (i.e. before this release) has just required I run headless Chrome in headed mode. Granted, I don't do a ton of scraping -- mostly just pulling data out of websites so that I can play with it in aggregate using more civilized tools.

[1]: https://github.com/berstend/puppeteer-extra/tree/master/pack...

Fortified Headless Browser: Depending on the anti-bot protection the website is using you make need to use a fortified headless browser that can solve its JS challenges without giving its identity away. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

That being said, if you're a beginner Selenium is a much more mature package so it has significantly more resources on StackOverflow and whatnot and Puppeteer has bigger community for avoiding web scraper detection (plugins like puppeteer-extra-plugin-stealth)

This extension takes an interesting approach to spoofing data, which is nice!

In my case, I’m interested in doing the same thing inside of Puppeteer for web scraping, unfortunately it seems like the only possible approach is similar to content scripts (for example https://github.com/berstend/puppeteer-extra/tree/master/pack...) which leads to it being easily detected. Are there any similar approaches that can be used for Puppeteer?

"I'm a noob and using python with selenium to do some basic scraping on StockX" and scraping protected website like stockx with perimeterx is not possible. It's all about reverse engineering, browser introspection, fingerprint (from hardware to software canvas), then you still need tons of ips to rotate and cooldown, finally protection evolve with time and you have to redo most of the things to pass again. A company like Scrapfly exists because it's more expensive to do and maintain such solution internally, look at their public repositories on GitHub low level stuff, network spoofing stacks, packet manipulation, custom angle libs. It takes a long time to learn vs something like `asp=true` from their docs https://scrapfly.io/docs/scrape-api/anti-scraping-protection If you have time and are more interested in this side, you could start to read https://github.com/prescience-data/dark-knowledge and look at https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth project to see how it works. Do not attempt stealth project helping you to bypass at scale, it's public, anti bot companies are aware and spot it easily - most of the time they don't block directly and use bad fp generated to recognize bots and map proxies ips to collect it and deducted the subnet or residential > My main question is, would it be better to try and make my script act "more human" It's a legend that anti bot use or detect "human" behavior, this signal is not very important, you can randomly move the mouse or things, like is fine, having 0 input events, is suspect but not that much in fact - tactile systems do not trigger any events until you touch so it can't be a strong signal due to false-positive - and doing "behavioral detection" is a big lie in the industry, you can experiment by doing dumb things, it's still passing and at scale ... and when they say "machine learning" it's just basic stats like a throttle do but based on browser fingerprints rather than IP. If you hit some path, like login, registration and payment - they can use some very heavy system with GPU canvas and stuff like but not used for scraping yet > are other methods like switching drivers and using proxies the way to go? Using proxies yes, but with wrong fingerprints (chrome headless, a browser running on server hardware, browser in docker and so on) In fact, there is no magic, mixing driver change nothing, they still manipulate a spotted browser - some are just more flexible than other to spoof correctly some part - like js worker interception to inject scripts and hook correctly but that's all.

The page scraped for this post behaves "interesting", sometimes the reCaptcha is ignored, some others appear right after submitting the login, so randomly fails; I opened an issue in puppeteer-extra, an npm lib extension for puppeteer which works hand-to-hand with 2captcha, I'm watching the issue closely, in case of getting a fix for the random issue I'll edit the post.

The stealth plugin for Puppeteer Extra gives a pretty good idea of what you need to cover today. Maybe it's not rocket science, but it's not trivial either.

https://github.com/berstend/puppeteer-extra/tree/master/pack...