colly VS chromedp

https://github.com/lib4u/fake-useragent https://github.com/gocolly/colly

Colly GitHub Repository

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Interesting. Can you compare it to colly? [0]

Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.

Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)

[0] https://github.com/gocolly/colly

Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

We could write some snippets mixing all these, but the best option in real life is to use a tool with it all, like Scrapy, pyspider, node-crawler (Node.js), or Colly (Go).

Unless you're specifically trying to do it using net/http, I recommend using colly. I've used it in a few scrappers and I love it!

chromedp lets you programmatically control Chrome.

To my surprise the code was closer in structure to the puppeteer version than it was to the chromedp. The chromedp version uses nested context declarations to manage the browser and page runtimes, the rust version uses a more linear approach. You construct a browser instance and then you can interact with it as a user would. This points at the fact that the chromiumoxide api is higher level.

Looking at the alternatives there are two that stand out chromedp and rod. Rod looks like it is the prodigal son of behave and cucumber some well established BDD frameworks. Personally I am not finding the MustYaddaYadda... very readable and combining it with other custom APIs would probably make it become inconsistent. It has a few nice things in the way it abstracts iframes but I am just unable to go past the higher level API.

There are a billion things that you need to consider when building a decent web crawler, especially interacting with pages in the modern web. For example, a lot of content is dynamically loaded by the browser nowadays, and won't show up if you make a simple HTTP request. Open your browser devtools and look at the network tab after you make a request, and you'll see it makes loads of auxiliary requests. Some content is also only loaded after you interact with it (e.g. hover, click). For that reason I'd recommend using something like chromedp and do browser based crawling, even if it's much slower.

You could use something like the chrome dev tools (https://github.com/chromedp/chromedp) or look into JS runtimes in Go and try to handle the I/O yourself.

Thanks. I've found Chromedp [0] to be a good Selenium replacement when programming in Go. Used in conjunction with HeadlessShell [1] you can deploy a Go app into a container and do the testing all within the same container with low overhead.

[0] https://github.com/chromedp/chromedp

Is it node specific? Could you just use https://github.com/chromedp/chromedp and run it as an actually webpage and get the value from that?

It looks like https://github.com/chromedp/chromedp is the most popular option. There's also https://github.com/playwright-community/playwright-go. Any thoughts or advice about using these tools?