Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →
Top 23 Go Crawler Projects
-
-
CodeRabbit
CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
-
https://github.com/lib4u/fake-useragent https://github.com/gocolly/colly
-
Project mention: Ask HN: How to find subdomains and paths for a website | news.ycombinator.com | 2024-06-01
-
crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
-
-
till
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
-
-
nebula
🌌 A network agnostic DHT crawler, monitor, and measurement tool that exposes timely information about DHT networks. (by dennis-tra)
-
-
-
-
dorkscout
DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets
-
-
-
-
spidy
Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)
-
node-crawler
Attempts to crawl the Ethereum network of valid Ethereum execution nodes and visualizes them in a nice web dashboard. (by ethereum)
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Go Crawler discussion
Go Crawler related posts
-
Golang with Colly: Use Random Fake User-Agents When Scraping
-
Wayurls
-
Intermediate Go Projects
-
Ask HN: How to find subdomains and paths for a website
-
Show HN: Nebula – A network agnostic DHT crawler
-
Modern automated data miner (scrapper)
-
Scraping the full snippet from Google search result
-
A note from our sponsor - CodeRabbit
coderabbit.ai | 24 Mar 2025
Index
What are some of the best open-source Crawler projects in Go? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | lux | 28,890 |
| 2 | colly | 23,896 |
| 3 | katana | 13,212 |
| 4 | crawlab | 11,670 |
| 5 | Pholcus | 7,581 |
| 6 | Ferret | 5,791 |
| 7 | crawlergo | 2,896 |
| 8 | Geziyor | 2,680 |
| 9 | cariddi | 1,636 |
| 10 | go-dork | 1,169 |
| 11 | till | 814 |
| 12 | webpalm | 362 |
| 13 | nebula | 335 |
| 14 | seonaut | 332 |
| 15 | crawley | 288 |
| 16 | antch | 262 |
| 17 | dorkscout | 234 |
| 18 | ChainWalker | 198 |
| 19 | slrp | 176 |
| 20 | gogetcrawl | 155 |
| 21 | spidy | 153 |
| 22 | node-crawler | 124 |
| 23 | scrapemate | 108 |
Sponsored
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai