colly
gofeed
Our great sponsors
colly | gofeed | |
---|---|---|
39 | 4 | |
22,165 | 2,454 | |
1.8% | - | |
6.0 | 6.1 | |
11 days ago | about 2 months ago | |
Go | Go | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
colly
-
Scraping the full snippet from Google search result
SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.
-
Show HN: Flyscrape – A standalone and scriptable web scraper in Go
Interesting. Can you compare it to colly? [0]
Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.
Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)
[0] https://github.com/gocolly/colly
- Colly: Elegant Scraper and Crawler Framework for Golang
-
New modern web crawling tool
Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?
-
colly VS scrapemate - a user suggested alternative
2 projects | 15 Apr 2023
-
Web Scraping in Python: Avoid Detection Like a Ninja
We could write some snippets mixing all these, but the best option in real life is to use a tool with it all, like Scrapy, pyspider, node-crawler (Node.js), or Colly (Go).
- Web scraping with Go
-
Web scraper help
Unless you're specifically trying to do it using net/http, I recommend using colly. I've used it in a few scrappers and I love it!
-
Web Scraping in Golang
In this blog, we will be covering the basics of web scraping in Go using the Fiber and Colly frameworks. Colly is an open-source web scraping framework written in Go. It provides a simple and flexible API for performing web scraping tasks, making it a popular choice among Go developers. Colly uses Go's concurrency features to efficiently handle multiple requests and extract data from websites. It offers a wide range of customization options, including the ability to set request headers, handle cookies, follow redirects, and more
-
Learn how to scrape Trustpilot reviews using Go
github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.
gofeed
-
IndieWebifying my Website Part 1 - Microformats and Webmentions
Luckily I did not have to implement any of this myself apart from some glue code to fit it together: I used the library gocron for scheduling the regular intervals, gofeed for parsing the RSS feed and webmention for extracting links and sending webmentions.
-
Show HN: The Brutalist Report – A rolling snapshot of the day’s headlines
The whole thing is written in Go on my end. Ingesting new headlines is handled in a goroutine that spawns within the process every 30 mins using a combo of the wonderful gofeed (https://github.com/mmcdole/gofeed) and colly (https://github.com/gocolly/colly) libraries.
When loading the front page, you're loading a 1-minute-cached HTML page of it that was constructed out of headlines already in my PostgreSQL database that were put there by the ingestion goroutine.
I like the idea of word clouds actually, I think you're on to something there. I think you just need to pre-generate them rather than doing it adhoc (if that's what you're doing here) for speed. Additionally, perhaps consider using sentiment in a way that orients stories based on positive and negative sentiment. Right now I am not seeing how I as a visitor/user can act on the sentiment analysis as it is presented now.
It would be neat to see a collection of uplifting stories grouped together through the sentiment analysis.
Anyway, food for thought. I hope you keep hacking away on it as it's just good fun to build things.
- Automatice el README para su perfil de GitHub con Go y GitHub Actions
-
Automate Your GitHub Profile README with Go and GitHub Actions
I needed to scan the blog feed and wanted to do it in Go, so the first thing I did was look for any libraries that would make it easier for me not to reinvent the wheel and I found the github.com/mmcdole/gofeed. It had a lot of features but I had enough with the basic use described in its README.
What are some alternatives?
GoQuery - A little like that j-thing, only in Go.
gographviz - Parses the Graphviz DOT language in golang
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
micro-editor - A modern and intuitive terminal-based text editor
xpath - XPath package for Golang, supports HTML, XML, JSON document query.
go-nmea - A NMEA parser library in pure Go
rod - A Devtools driver for web automation and scraping
ODF - Open Document Format (ODF) generator library for Go.
Geziyor - Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
go-pkg-rss
Ferret - Declarative web scraping
xml - Package feed implements a flexible, robust and efficient RSS and Atom parser