Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 12 Go Spider Projects
-
SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.
-
crawlab
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
Project mention: Show HN: I scraped 25M Shopify products to build a search engine | news.ycombinator.com | 2023-12-13
As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.
-
cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
spidy
Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)
-
Project mention: colly VS scrapemate - a user suggested alternative | libhunt.com/r/colly | 2023-04-15
-
Go Spider related posts
- New modern web crawling tool
-
DHT VS dht - a user suggested alternative
2 projects | 13 Jan 2022
- Show HN: A fast, feature-rich crawler for Go
- Fast, feature-rich web crawler for Go
- Feature rich crawler for Go.
- Create a tiny crawler/scraper for Go
-
A note from our sponsor - InfluxDB
www.influxdata.com | 28 Mar 2024
Index
What are some of the best open-source Spider projects in Go? This list will help you:
Project | Stars | |
---|---|---|
1 | colly | 21,939 |
2 | crawlab | 10,700 |
3 | Pholcus | 7,504 |
4 | DHT | 2,668 |
5 | Geziyor | 2,464 |
6 | cariddi | 1,327 |
7 | webpalm | 325 |
8 | ant | 276 |
9 | gospider | 203 |
10 | spidy | 138 |
11 | scrapemate | 47 |
12 | turbo-tor-crawl | 6 |