SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Go Scraper Projects
-
Not a fix, but I tend to use lux when downloading from bilibili. It is faster too.
-
Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
-
-
-
till
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
-
mangal
📖 The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!
-
InfluxDB
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
-
-
-
-
GMDB
GMDB is the ultra-simple, cross-platform Movie Library with Features (Search, Take Note, Watch Later, Like, Import, Learn, Instantly Torrent Magnet Watch)
-
dorkscout
DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets
-
-
meteor
Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog. (by odpf)
Project mention: Modern open-source data platform that empowers organizations to discover, transform, analyse and secure data faster and efficiently. | /r/dataengineering | 2022-06-21Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.
-
Project mention: scraply v3.0.0 - a very simple scraping too using CSS selectors and jQuery like functions | /r/golang | 2022-07-06
-
spidy
Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)
Project mention: Share Your Code.. Share your most unique piece of Go code. | /r/golang | 2022-10-151 - Expired domain scrapper => https://github.com/twiny/spidy 2 - A sample & efficient web crawler => https://github.com/twiny/wbot 3 - A mini blockchain scanner => https://github.com/twiny/blockscan 4 - A Snake Game => https://github.com/twiny/snaky
-
Hey, I am also relatively new to Go The project I’m maintaining is Grab
-
Project mention: Command line tool to bulk download images from reddit | /r/commandline | 2023-02-08
-
Project mention: I created Prometheus Exporter with Go to scrape my xDSL Modem stats | /r/golang | 2023-03-17
-
-
-
I am currently building a webscraper, called goskyr, that can be run from the command line and is supposed to be easily configurable. So instead of having to write code to scrape a website you'd just write a configuration snippet and run the scraper. I realize that there are a number of gui based scraping services that make it extremely easy to setup a scraping process for any website, so for people having no coding experience whatsoever that would probably be the easiest solution. I'm trying to come close to those gui based solutions in terms of functionality by providing a 'smart' way of finding potentially interesting data/fields and letting the user select a subset in a terminal based ui. Also date extraction & parsing and the newly added machine learning capability is probably worth mentioning. Still, those other, gui based solutions are really awesome, eg octoparse or scrapestorm.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-05-22.
Go Scraper related posts
- Bilibili download stalls at around 30-60%
-
colly VS scrapemate - a user suggested alternative
2 projects | 15 Apr 2023
- Web Scraping in Python: Avoid Detection Like a Ninja
- Web scraping with Go
- Show HN: Library for scrape internet like GQL
- No code command line webscraper
- Fitter - GQL for web scrawling, multiple sources, defining fields, emulating browser and more other
-
A note from our sponsor - #<SponsorshipServiceOld:0x00007f0922f73d48>
www.saashub.com | 5 Jun 2023
Index
What are some of the best open-source Scraper projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | lux | 21,142 |
| 2 | colly | 19,649 |
| 3 | Ferret | 5,366 |
| 4 | rod | 3,836 |
| 5 | Geziyor | 1,932 |
| 6 | till | 799 |
| 7 | mangal | 685 |
| 8 | finance-go | 601 |
| 9 | Dataflow kit | 593 |
| 10 | ant | 273 |
| 11 | GMDB | 226 |
| 12 | dorkscout | 199 |
| 13 | demeter | 163 |
| 14 | meteor | 145 |
| 15 | scraply | 123 |
| 16 | spidy | 116 |
| 17 | grab | 63 |
| 18 | rrip | 56 |
| 19 | xdsl-exporter | 44 |
| 20 | fitter | 40 |
| 21 | go-recipe | 20 |
| 22 | goskyr | 19 |
| 23 | moviestills | 13 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com