Avoiding bot detection: How to scrape the web without getting blocked?

1. browser-fingerprinting

   1 8 4,993 2.4 JavaScript

   Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

   

   Try your technique on a few of these fingerprint testing sites https://github.com/niespodd/browser-fingerprinting#fingerpri... I'm pretty sure it's quite detectible

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. duo-bypass

   2 5 128 3.5 Python

   Stuff after reverse engineering DUO's mobile app.

   

   There are myriad ways of extracting the TOTP seed from these apps... Or you just reverse engineer the setup/confirmation process and then you can generate/trigger your own tokens from your automation workflow.

   2FA is a good security feature but it does not help against web scraping. Credential stuffing and other 3rd party attacks? Yes, it _can_ help. But it does not always help. There's a phishing group that has seemingly specialised on getting people to click the green confirm button in their Duo app... ¯\_(ツ)_/¯

   Check https://github.com/revalo/duo-bypass for a python script that can be used to automate Duo tokens... Has some code from me. There are similar scripts for all the other well known OTP Apps...

4. quaid

   3 3 16 10.0 JavaScript

   A single-page webapp that decrypts text using only client-side JavaScript

   

   This utility will help with that, assuming the services that use 2FA have a backup-code feature: https://github.com/sowbug/quaid

5. undetected-chromedriver

   4 43 12,670 1.8 Python

   Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

   

   I've had a lot of success just with Selenium and this custom version of Chromedriver: https://github.com/ultrafunkamsterdam/undetected-chromedrive...

6. undetected-chromedrive

   5 2 - -

   I've had a lot of success just with Selenium and this custom version of Chromedriver: https://github.com/ultrafunkamsterdam/undetected-chromedrive...

7. Playwright

   6 537 90,281 10.0 TypeScript

   Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

   

   Playwright is easy to get started with. The even tools that allow you to record your browser actions and covert it into code ( https://playwright.dev/ ).

Suggest a related project