Go Crawler

Open-source Go projects categorized as Crawler | Edit details

Top 19 Go Crawler Projects

  • lux

    👾 Fast and simple video download library and CLI tool written in Go

    Project mention: Annie: Simple video download CLI tool like YouTube-dl written in Go | reddit.com/r/CKsTechNews | 2021-12-19
  • colly

    Elegant Scraper and Crawler Framework for Golang

    Project mention: Web Crawling Libraries which allow me to view the network requests made from a site? | reddit.com/r/golang | 2021-12-16

    I've been looking at various scraping/crawling libraries like colly, but I'm not exactly sure if any of them can handle a specific use case I have.

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • Pholcus

    Pholcus is a distributed high-concurrency crawler software written in pure golang

  • Ferret

    Declarative web scraping

  • Rendora

    dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

    Project mention: How to make Svelte play nice with backend and SEO | reddit.com/r/sveltejs | 2021-12-14

    keep the strategy above, and use https://github.com/rendora/rendora but not sure if it would play nice with SEO/robots (haven't tried)

  • Geziyor

    Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

  • go-dork

    The fastest dork scanner written in Go.

    Project mention: go-dork: El escáner dork más rápido escrito en Go 🌩 | reddit.com/r/u_esgeeks | 2021-04-05
  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • cariddi

    Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more...

    Project mention: cariddi released! | dev.to | 2021-06-13

    https://github.com/edoardottt/cariddi Take a list of domains and scan for endpoints, secrets, api keys, file extensions, tokens and more...

  • antch

    Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  • dorkscout

    DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

    Project mention: automatic google dorking scan tool | reddit.com/r/AccessCyber | 2021-08-17
  • bathyscaphe

    Fast, highly configurable, cloud native dark web crawler.

  • crawley

    The unix-way web crawler (by s0rg)

    Project mention: crawley - the unix-way web-crawler | reddit.com/r/commandline | 2021-11-10

    https://github.com/s0rg/crawley/releases/tag/v1.1.6 is online and have "-dirs" option, just for this task.

  • pagser

    Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

  • skweez

    Fast website scraper and wordlist generator

    Project mention: Skweez - fast domain crawler and wordlist generator | reddit.com/r/redteamsec | 2021-11-14
  • google-search-results-golang

    Google Search Results GoLang API

  • gichidan

    Gichidan - CLI wrapper for Ichidan deep-web search engine.

  • ytwrk

    YT metadata exporter

    Project mention: How can I archive YouTube comments? | reddit.com/r/DataHoarder | 2021-03-28

    There's also https://github.com/terorie/ytwrk

  • jsonrpconion

    Library for building JSON RPC services on Tor network

  • gopixabay

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-12-19.

Go Crawler related posts


What are some of the best open-source Crawler projects in Go? This list will help you:

Project Stars
1 lux 16,847
2 colly 15,821
3 Pholcus 7,029
4 Ferret 4,859
5 Rendora 1,846
6 Geziyor 1,614
7 go-dork 506
8 cariddi 358
9 antch 221
10 dorkscout 157
11 bathyscaphe 70
12 crawley 45
13 pagser 42
14 skweez 42
15 google-search-results-golang 34
16 gichidan 29
17 ytwrk 29
18 jsonrpconion 1
19 gopixabay 0
Find remote jobs at our new job board 99remotejobs.com. There are 30 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
OPS - Build and Run Open Source Unikernels
Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.