puppeteer-extra VS node

You could use https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth A plugin to escape anti bot detection

There is a good chance that the website is using Cloudflare to block web scrapers, which will require you to use a fortified headless browser to solve the JS challenges. Your options include the Puppeteer stealth plugin and Selenium undetected-chromedriver.

There are even Puppeteer plugins that will do it for you. [^1]

The best detection I've come across so far (i.e. before this release) has just required I run headless Chrome in headed mode. Granted, I don't do a ton of scraping -- mostly just pulling data out of websites so that I can play with it in aggregate using more civilized tools.

[1]: https://github.com/berstend/puppeteer-extra/tree/master/pack...

You didn't know? https://github.com/berstend/puppeteer-extra/wiki/Block-resources-without-request-interception

As for solutions, you are on point. Running a headless browser or using a web scraping API that does that for you (I work at one: https://scrapfly.io hi) is the easiest way to do it. Note that because of javascript fingerprinting you still need to fortify your headless browsers with various scripts like puppeteer-stealth.

Unfortunately, it's now trivial to bypass recaptcha: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-recaptcha

Use a fortified headless browser like the stealth plugin for puppeteer.

To get started with Node.js 22.0.0, you can download the latest version from the official Node.js website. For those upgrading from an older version, it is recommended to test your existing applications for compatibility with the new release, particularly if you are using native modules or rely heavily on third-party npm packages.

Node.js (version 10 or higher)

Before we start, ensure that you have Node.js installed on your computer. This is necessary because we will use Node's package manager (npm) to create our Next.js project. If you need to install Node.js, you can download it from the official Node.js website.

Before starting a new project in react, you need to make sure that you have NodeJS install on your system. You can download the latest version of node at https://nodejs.org. Follow the instructions on the node website to do the installation.

Installing Node.js: Download and install Node.js from nodejs.org. Choose the version recommended for most users, unless you have specific needs that require the latest features or earlier compatibility.

You should have Nodejs installed on your laptop and if not, check the Node.js official website, and download/ install the latest and stable release.

[Nest] 60017 - 04/22/2024, 1:07:48 PM ERROR Error [ERR_INTERNAL_ASSERTION]: Error: BSONError: Cannot create Buffer from undefined at Object.toLocalBufferType at Object.toHex at ObjectId.toHexString at ObjectId.inspect at ObjectId.[nodejs.util.inspect.custom] at formatValue (node:internal/util/inspect:782:19) at formatProperty (node:internal/util/inspect:1819:11) at formatArray (node:internal/util/inspect:1645:17) at formatRaw (node:internal/util/inspect:1027:14) at formatValue (node:internal/util/inspect:817:10) This is caused by either a bug in Node.js or incorrect usage of Node.js internals. Please open an issue with this stack trace at https://github.com/nodejs/node/issues

Node.js