Top 12 Go Scraping Projects

colly

10 21,202 0.0 Go

Elegant Scraper and Crawler Framework for Golang

Project mention: Show HN: Flyscrape – A standalone and scriptable web scraper in Go | news.ycombinator.com | 2023-11-11

Interesting. Can you compare it to colly? [0]
Last time I looked it was the most popular choice for scraping in Go and I have some projects using it.
Is it similar? Does it have more/less features or is it more suited for a different use case? (Which one?)
[0] https://github.com/gocolly/colly
Ferret

0 5,509 0.0 Go

Declarative web scraping
InfluxDB

www.influxdata.com
sponsored

Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
Geziyor

1 2,294 0.0 Go

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Project mention: Show HN: Flyscrape – A standalone and scriptable web scraper in Go | news.ycombinator.com | 2023-11-11

Its been 8+ years since i started scraping. I even wrote a popular Go web scraping framework previously: (https://github.com/geziyor/geziyor).
These days, I'm not even using Go for scraping, as the webpage changes makes me crazy, so I moved to Typescript+Playwright. (Crawlee framework is cool, while not strictly necessary).
My favorite stack as of 2023: TypeScript+Playwright+Crawlee(Optional)
till

0 803 1.8 Go

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Dataflow kit

0 623 0.0 Go

Extract structured data from web sites. Web sites scraping.
antch

0 252 0.0 Go

Antch, a fast, powerful and extensible web crawling & scraping framework for Go
newser

0 83 3.2 Go

Newser is a simple utility to generate a pdf with you favorite news articles
Onboard AI

getonboard.dev
sponsored

Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.
goclone

1 45 10.0 Go

🌱 goclone - clone websites in a matter of seconds

Project mention: Show HN: Goclone – your ultimate tool for offline web browsing | news.ycombinator.com | 2023-07-31
xdsl-exporter

3 45 10.0 Go

xDSL Prometheus Exporter

Project mention: I created Prometheus Exporter with Go to scrape my xDSL Modem stats | /r/golang | 2023-03-17
goGetJS

0 30 0.0 Go

a tool for extracting, searching, and saving JavaScript files (with optional headless browser)
moviestills

0 13 0.0 Go

A small CLI app to scrap high-quality movie snapshots from various websites.
go-scraper

1 0 10.0 Go

This repo show how to Scrape different type of data

Project mention: How to scrape different types of data in Golang Using Colly | dev.to | 2023-01-29

GitHub Link

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-11-11.