lambdasoup VS colly

I'm using soupault right now to make a simple company wiki (under a dozen pages). I like how it's HTML-first and easily customizable, compared to other static site generators that come with too many bells and whistles. Although now I have to make my own image compression script...

Also, fun fact: soupault is written in OCaml, which apparently has a really nice library for HTML manipulation: https://github.com/aantron/lambdasoup

OCaml’s Lambda Soup (https://aantron.github.io/lambdasoup/) is a amazing library/, especially for those that prefer functional programming

It's used for sorting "widgets" (page processing steps) according to dependency lists that users can specify in the config (like `after = ["foo", "bar"]`).

Other than that, one thing I really like about OCaml is that the compiler team and most library maintainers are considerate towards downstream users with respect to compatibility.

The Lua interpreter [3] that soupault uses for its plugin API is a revived 20 year old research project. It only needed minor modifications to build with recent compiler versions.

[1] https://github.com/aantron/lambdasoup

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Interesting. Can you compare it to colly? [0]

Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.

Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)

[0] https://github.com/gocolly/colly

Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

We could write some snippets mixing all these, but the best option in real life is to use a tool with it all, like Scrapy, pyspider, node-crawler (Node.js), or Colly (Go).

Unless you're specifically trying to do it using net/http, I recommend using colly. I've used it in a few scrappers and I love it!

In this blog, we will be covering the basics of web scraping in Go using the Fiber and Colly frameworks. Colly is an open-source web scraping framework written in Go. It provides a simple and flexible API for performing web scraping tasks, making it a popular choice among Go developers. Colly uses Go's concurrency features to efficiently handle multiple requests and extract data from websites. It offers a wide range of customization options, including the ability to set request headers, handle cookies, follow redirects, and more

github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.