puppeteer-extra VS frida

You could use https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth A plugin to escape anti bot detection

There is a good chance that the website is using Cloudflare to block web scrapers, which will require you to use a fortified headless browser to solve the JS challenges. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

There are even Puppeteer plugins that will do it for you. [^1]

The best detection I've come across so far (i.e. before this release) has just required I run headless Chrome in headed mode. Granted, I don't do a ton of scraping -- mostly just pulling data out of websites so that I can play with it in aggregate using more civilized tools.

[1]: https://github.com/berstend/puppeteer-extra/tree/master/pack...

You didn't know? https://github.com/berstend/puppeteer-extra/wiki/Block-resources-without-request-interception

As for solutions, you are on point. Running a headless browser or using a web scraping API that does that for you (I work at one: https://scrapfly.io hi) is the easiest way to do it. Note that because of javascript fingerprinting you still need to fortify your headless browsers with various scripts like puppeteer-stealth.

Unfortunately, it's now trivial to bypass recaptcha: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-recaptcha

Use a fortified headless browser like the stealth plugin for puppeteer.

Frida, uff this is just AMAZING, yes with uppercase and in bold letters. They also has bindings on different languages that can be found in their github repository. Spoiler alert...the Go binding it's pure shit...really couldn't run it. Use just the default that it's installed with pip install frida-tools.

A great framework for doing something along those lines is Frida (https://github.com/frida/frida). Works on a bunch of stuff, including Android and iOS. Some global-ish certificate pinning bypasses work through Frida, by patching http libraries to not raise exceptions, accept system certificates, etc and just quietly hum along instead. Certificate unpinning in turn enables network MITM with mitmproxy, which makes it a lot quicker and easier to inspect, block, or modify network traffic.

Funnily enough, I've seen much stronger obfuscation from reverse engineering from my cheap Tuya IoT devices app than from my bank app.

Install Frida from Github :- https://github.com/frida/frida

// see: https://github.com/frida/frida/issues/382

If anyone needs a "monkey" not for web pages but for any process on your computer system, may I recommend Frida:

https://frida.re

https://github.com/frida/frida

With Frida, you write JavaScript programs and inject them into arbitrary processes, to hook and modify and call whatever you please.

It gets a lot of use in the reverse engineering and vulnerability research communities, but has broader scope too. For instance, I used it recently to automate the UI of a video production program on Windows, by injecting a thread that sends window messages to the main message loop and hooks into various system dialog functions.

var android_log_write = new NativeFunction( Module.getExportByName(null, '__android_log_write'), 'int', ['int', 'pointer', 'pointer'] ); var tag = Memory.allocUtf8String("[frida-sript][ax]"); var work = function() { setTimeout(function() { android_log_write(3, tag, Memory.allocUtf8String("ping @ " + Date.now())); work(); }, 1000); } work(); // console.log does not seems to work. see: https://github.com/frida/frida/issues/382 console.log("console.log"); console.error("console.error"); console.warn("WARN"); android_log_write(3, tag, Memory.allocUtf8String(">--(O.o)-<)");

Go to https://github.com/frida/frida/releases and download the latest frida-server--android-arm64.xz. Extract it and run adb push frida-server--android-arm64 /sdcard/frida-server

It sounds like a kind of black magic:

> ...It’s a dynamic code instrumentation toolkit. It lets you inject snippets of JavaScript or your own library into native apps on Windows, macOS, GNU/Linux, iOS, Android, and QNX.

> ...Frida’s core is written in C and injects QuickJS into the target processes, where your JS gets executed with full access to memory, hooking functions and even calling native functions inside the process.

> There’s a bi-directional communication channel that is used to talk between your app and the JS running inside the target process.

Here's a description of the architecture:

https://frida.re/docs/hacking/

And the source:

https://github.com/frida/frida

---

Apparently using "wxWindows Library Licence, Version 3.1":

> This is essentially the LGPL, with an exception stating that derived works in binary form may be distributed on the user's own terms. This is a solution that satisfies those who wish to produce GPL'ed software using Frida, and also those producing proprietary software.

https://github.com/frida/frida/blob/master/COPYING