colly VS GoQuery

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Interesting. Can you compare it to colly? [0]

Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.

Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)

[0] https://github.com/gocolly/colly

Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

We could write some snippets mixing all these, but the best option in real life is to use a tool with it all, like Scrapy, pyspider, node-crawler (Node.js), or Colly (Go).

Unless you're specifically trying to do it using net/http, I recommend using colly. I've used it in a few scrappers and I love it!

In this blog, we will be covering the basics of web scraping in Go using the Fiber and Colly frameworks. Colly is an open-source web scraping framework written in Go. It provides a simple and flexible API for performing web scraping tasks, making it a popular choice among Go developers. Colly uses Go's concurrency features to efficiently handle multiple requests and extract data from websites. It offers a wide range of customization options, including the ability to set request headers, handle cookies, follow redirects, and more

github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.

Your comment was posted 4 minutes ago. That means you still have enough time to edit your comment to change it so it contains real URLs:

<https://github.com/PuerkitoBio/goquery>

<https://github.com/dop251/goja>

(Please do not reply to this comment—I won't be able to delete it once the previous post is fixed if it contains replies.)

How about - https://github.com/PuerkitoBio/goquery

Assuming you're asking about https://github.com/PuerkitoBio/goquery , to interpret your printout you want to look at what Find is defined to return, a *Selection. https://github.com/PuerkitoBio/goquery/blob/39fb6d4dc47a07e5782494b6defc89a194b1f906/traversal.go#L23

github.com/PuerkitoBio/goquery - library that provides a convenient and concise way to query HTML and XML documents. It provides a jQuery-like API for selecting elements and extracting data, making it a popular choice for web scraping in Go.

Yep, I thing I can add it. Add new goquery alternative will be good :)

If I want to parse the structure of HTML and not interact with it from a browser point of view, I use this in Go: https://github.com/PuerkitoBio/goquery

goquery

I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM.

Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site.

https://github.com/PuerkitoBio/goquery

If you're willing to add constraints around the goal, you can catch this type of error with semgrep rules and/or unit tests using qoquery.

RUN go get github.com/PuerkitoBio/goquery