Go Spider

Open-source Go projects categorized as Spider

Top 12 Go Spider Projects

  • colly

    Elegant Scraper and Crawler Framework for Golang

    Project mention: Scraping the full snippet from Google search result | dev.to | 2024-01-01

    SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

  • crawlab

    Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • Pholcus

    Pholcus is a distributed high-concurrency crawler software written in pure golang

  • DHT

    BitTorrent DHT Protocol && DHT Spider.

  • Geziyor

    Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

    Project mention: Show HN: I scraped 25M Shopify products to build a search engine | news.ycombinator.com | 2023-12-13

    As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.

  • cariddi

    Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  • webpalm

    🕸️ Crawl in the web network

    Project mention: Modern automated data miner (scrapper) | news.ycombinator.com | 2024-02-08
  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • ant

    A web crawler for Go (by yields)

  • gospider

    ⚡ Light weight Golang spider framework | 轻量的 Golang 爬虫框架

  • spidy

    Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)

  • scrapemate

    Golang Crawling and scraping framework (by gosom)

    Project mention: colly VS scrapemate - a user suggested alternative | libhunt.com/r/colly | 2023-04-15
  • turbo-tor-crawl

    Recursive hostnames crawler

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-08.

Go Spider related posts

Index

What are some of the best open-source Spider projects in Go? This list will help you:

Project Stars
1 colly 21,939
2 crawlab 10,700
3 Pholcus 7,504
4 DHT 2,668
5 Geziyor 2,464
6 cariddi 1,327
7 webpalm 325
8 ant 276
9 gospider 203
10 spidy 138
11 scrapemate 47
12 turbo-tor-crawl 6
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com