Top 12 Go Spider Projects

Spider

• Add a project

• colly

  40 22,970 5.0 Go

  Elegant Scraper and Crawler Framework for Golang

  

  Project mention: Intermediate Go Projects | dev.to | 2024-06-10

  Colly GitHub Repository

• InfluxDB

  www.influxdata.com featured

  Purpose built for real-time analytics at any scale. InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

  

• crawlab

  4 11,212 5.5 Go

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  

• Pholcus

  0 7,558 0.0 Go

  Pholcus is a distributed high-concurrency crawler software written in pure golang

• DHT

  1 2,716 0.0 Go

  BitTorrent DHT Protocol && DHT Spider.

• Geziyor

  2 2,575 2.6 Go

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  

  Project mention: Show HN: I scraped 25M Shopify products to build a search engine | news.ycombinator.com | 2023-12-13

  As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.

• webpalm

  11 348 7.8 Go

  🕸️ Crawl in the web network

  

  Project mention: Modern automated data miner (scrapper) | news.ycombinator.com | 2024-02-08

• ant

  4 277 0.0 Go

  A web crawler for Go (by yields)

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• gospider

  0 203 3.6 Go

  ⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

• spidy

  4 147 0.0 Go

  Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)

• scrapemate

  1 76 4.0 Go

  Golang Crawling and scraping framework (by gosom)

• wbot

  2 17 4.6 Go

  A simple & efficient web crawler.

• turbo-tor-crawl

  1 5 10.0 Go

  Recursive hostnames crawler

Go Spider related posts

• New modern web crawling tool

  2 projects | news.ycombinator.com | 30 Apr 2023

• DHT VS dht - a user suggested alternative

  2 projects | 13 Jan 2022

• Show HN: A fast, feature-rich crawler for Go

  1 project | news.ycombinator.com | 26 Mar 2021

• Fast, feature-rich web crawler for Go

  1 project | /r/golang | 26 Mar 2021

• Feature rich crawler for Go.

  1 project | /r/golang | 26 Mar 2021

• Create a tiny crawler/scraper for Go

  1 project | /r/golang | 26 Mar 2021
• A note from our sponsor - InfluxDB
  www.influxdata.com | 9 Sep 2024

  InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards. Learn more →

Index

Sponsored

Purpose built for real-time analytics at any scale.

InfluxDB Platform is powered by columnar analytics, optimized for cost-efficient storage, and built with open data standards.

www.influxdata.com

Do not miss the trending Go projects with our weekly report!