chrome-aws-lambda VS colly

Lambda would be my choice for this. You could even stay within the free tier depending how often you run this process. You can orchestrate puppeteer UI flows in lambda using this package https://github.com/alixaxel/chrome-aws-lambda My team does this and it works great.

git clone --depth=1 https://github.com/alixaxel/chrome-aws-lambda.git && \ cd chrome-aws-lambda && \ make chrome_aws_lambda.zip

When requesting the API route, the Next.js serverless function will actually spin up a web browser on the server (a headless instance of Chromium, using chrome-aws-lambda). Next, a webpage will be generated with HTML we can define ourselves. This HTML will be used to construct the image. That means that as a developer we can generate images using HTML and CSS, technologies we are already familiar with!

This plugin is used for all our AWS Lambda deployments, using a wide range of Node modules, some with more quirks than others. We use it together with Lambda Layer Sharp and Chrome AWS Lambda.

I was able to successfully to run chrome with puppeteer in AWS Lambda for a similar use case. I used an "optimized" version of chrome packaged as an AWS Lambda Layer.

git clone --depth=1 https://github.com/alixaxel/chrome-aws-lambda.git && \ cd chrome-aws-lambda && \ make chrome_aws_lambda.zip

Simplest method use ]Puppeteer](https://blog.risingstack.com/pdf-from-html-node-js-puppeteer/) with chrome-aws-lambda.

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Interesting. Can you compare it to colly? [0]

Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.

Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)

[0] https://github.com/gocolly/colly

Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

We could write some snippets mixing all these, but the best option in real life is to use a tool with it all, like Scrapy, pyspider, node-crawler (Node.js), or Colly (Go).

Unless you're specifically trying to do it using net/http, I recommend using colly. I've used it in a few scrappers and I love it!

In this blog, we will be covering the basics of web scraping in Go using the Fiber and Colly frameworks. Colly is an open-source web scraping framework written in Go. It provides a simple and flexible API for performing web scraping tasks, making it a popular choice among Go developers. Colly uses Go's concurrency features to efficiently handle multiple requests and extract data from websites. It offers a wide range of customization options, including the ability to set request headers, handle cookies, follow redirects, and more

github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.