puppeteer-cluster VS puppeteer

To make Unlighthouse fast, we combined this lighthouse binary with the package puppeteer-cluster, which allows for multi-threaded lighthouse scans.

https://github.com/thomasdondorf/puppeteer-cluster is doing queue-like mechanism but it is not using redis.

I've used this library: Puppeteer cluster

Literally just went to a Github Project (puppeteer in our case) and sorted issues by most comments/reactions. The number one was "How to run on Ubunutu" or something similar. Actually, it's this issue right here: https://github.com/puppeteer/puppeteer/issues/290.

Motivated by this issue https://github.com/puppeteer/puppeteer/issues/5893

It is well-known, old like a crap issue, I took the link from GitHub issue page https://github.com/puppeteer/puppeteer/issues/5893

First, we need to create a Node.js* project and add npm packages puppeteer, puppeteer-extra and puppeteer-extra-plugin-stealth to control Chromium (or Chrome, or Firefox, but now we work only with Chromium which is used by default) over the DevTools Protocol in headless or non-headless mode.

You can load extensions in puppeteer, also in headless with the experimental chrome mode. https://github.com/puppeteer/puppeteer/blob/main/docs/api.md...

It's really not that hard. I've used Puppeteer (https://www.npmjs.com/package/puppeteer) in the past for similar projects. If you don't have any programming background, search for third party scraping tools with a built-in rotating proxy service - there are a few about, just give it a Google.

After some research and understanding of what we need to design and build, we chose Docker Compose, RPyC, and Puppeteer for our test automation suite. The tech stack for our E2E testing is completely independent of the stack we chose for our product of course (we’ll dive into this in a separate post), and it can run with any product that is able to run Docker and Docker Compose in their environment.

I'd recommend using Puppeteer, which can drive Chrome in headful or headless modes. In dev mode, you can spin up a headful browser so that you can watch what your script is doing and debug it. In production, you'll just flip the condition to headless.

// File: pages/api/index.js // This is accessible from the deployed link (say, Y.com) as y.com/api?queryparametershere import core from 'puppeteer-core' import chromium from 'chrome-aws-lambda' export default async function handler(req, res) { // Only allow POST to the given route if (req.method === 'GET') { const { title, mode, image, width = 1400, height = 720 } = req.query // Launching chrome with puppeteer-core // https://github.com/puppeteer/puppeteer/issues/3543#issuecomment-438835878 const browser = await core.launch({ args: chromium.args, defaultViewport: chromium.defaultViewport, executablePath: await chromium.executablePath, headless: chromium.headless, ignoreHTTPSErrors: true, }) // Create a page const page = await browser.newPage() // Define the dimensions of the page await page.setViewport({ width: parseInt(width), height: parseInt(height) }) // Load the /dynamic_blogs with the given query paramters // Don't forget to encode them! // req.headers.host allows to obtain the deployed link as is, hence this app can be deployed anywhere // This allows us to take advantage of Layer0 caching to serve the /dynamic_blogs pages faster to this .goto() call await page.goto(`https://${req.headers.host}/dynamic_blogs?title=${encodeURIComponent(title)}&image=${encodeURIComponent(image)}&mode=${encodeURIComponent(mode)}`) // On average, place an image that is fast to load. // Falling back to 5 seconds timeout where image might take longer to load. await page.waitForTimeout(5000) // Take screenshot of the body of the page, that is the content const content = await page.$('body') const imageBuffer = await content.screenshot({ omitBackground: true }) await page.close() await browser.close() res.setHeader('Cache-Control', 'public, immutable, no-transform, s-maxage=31536000, max-age=31536000') res.setHeader('Content-Type', 'image/png') res.send(imageBuffer) res.status(200) return } // Any other method than GET results in a ERROR 400. res.status(400).json({ message: 'Invalid method.' }) return }

Here's the kicker, all we are going to use is one script and under 50 lines of code. How are we going to do this you may ask? The magic of JavaScript. Furthermore, a node library called Puppeteer and a web service called Browserless. If you don't know much about these tools I'd highly recommend you check them out further. They offer a lot of neat capabilities! We will go over some of them in this article.