Go Crawler

Open-source Go projects categorized as Crawler

Top 23 Go Crawler Projects

  • lux

    👾 Fast and simple video download library and CLI tool written in Go

  • colly

    Elegant Scraper and Crawler Framework for Golang

  • Project mention: Scraping the full snippet from Google search result | dev.to | 2024-01-01

    SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • crawlab

    Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

  • katana

    A next-generation crawling and spidering framework.

  • Pholcus

    Pholcus is a distributed high-concurrency crawler software written in pure golang

  • Ferret

    Declarative web scraping

  • crawlergo

    A powerful browser crawler for web vulnerability scanners

  • Project mention: Ethical Hacking Tool | /r/hackthebox | 2023-06-27
  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  • Geziyor

    Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  • Project mention: Show HN: I scraped 25M Shopify products to build a search engine | news.ycombinator.com | 2023-12-13

    As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.

  • Rendora

    dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

  • cariddi

    Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  • go-dork

    The fastest dork scanner written in Go.

  • till

    DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

  • webpalm

    🕸️ Crawl in the web network

  • Project mention: Modern automated data miner (scrapper) | news.ycombinator.com | 2024-02-08
  • nebula

    🌌 A network agnostic DHT crawler, monitor, and measurement tool that exposes timely information about DHT networks. (by dennis-tra)

  • Project mention: Show HN: Nebula – A network agnostic DHT crawler | news.ycombinator.com | 2024-03-20
  • antch

    Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  • crawley

    The unix-way web crawler (by s0rg)

  • dorkscout

    DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

  • ChainWalker

    Rapid Smart Contract Crawler

  • seonaut

    Open source SEO auditing tool.

  • slrp

    rotating open proxy multiplexer

  • spidy

    Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)

  • gogetcrawl

    Extract web archive data using Wayback Machine and Common Crawl

  • Project mention: A tool/package for Web Archive data extraction | /r/golang | 2023-05-31

    I've developed yet another solution that can help you extract data from web archives :) You can use it as a separate tool, or import it into your Go project. Github: https://github.com/karust/gogetcrawl

  • node-crawler

    Attempts to crawl the Ethereum network of valid Ethereum execution nodes and visualizes them in a nice web dashboard. (by ethereum)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Go Crawler related posts

  • Show HN: Nebula – A network agnostic DHT crawler

    1 project | news.ycombinator.com | 20 Mar 2024
  • Modern automated data miner (scrapper)

    1 project | news.ycombinator.com | 8 Feb 2024
  • Scraping the full snippet from Google search result

    3 projects | dev.to | 1 Jan 2024
  • Show HN: I scraped 25M Shopify products to build a search engine

    4 projects | news.ycombinator.com | 13 Dec 2023
  • Show HN: Flyscrape – A standalone and scriptable web scraper in Go

    6 projects | news.ycombinator.com | 11 Nov 2023
  • New webcrawler for bug-hunters and data-miners

    1 project | news.ycombinator.com | 18 Oct 2023
  • Colly: Elegant Scraper and Crawler Framework for Golang

    1 project | news.ycombinator.com | 23 Aug 2023
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 28 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

What are some of the best open-source Crawler projects in Go? This list will help you:

Project Stars
1 lux 25,624
2 colly 22,333
3 crawlab 10,890
4 katana 8,857
5 Pholcus 7,533
6 Ferret 5,641
7 crawlergo 2,770
8 Geziyor 2,492
9 Rendora 1,997
10 cariddi 1,376
11 go-dork 1,008
12 till 807
13 webpalm 328
14 nebula 282
15 antch 257
16 crawley 233
17 dorkscout 223
18 ChainWalker 192
19 seonaut 160
20 slrp 151
21 spidy 142
22 gogetcrawl 132
23 node-crawler 110

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com