Go Crawling

Open-source Go projects categorized as Crawling

Top 9 Go Crawling Projects

  • colly

    Elegant Scraper and Crawler Framework for Golang

    Project mention: Colly: Elegant Scraper and Crawler Framework for Golang | news.ycombinator.com | 2023-08-23
  • Ferret

    Declarative web scraping

  • Mergify

    Tired of breaking your main and manually rebasing outdated pull requests?. Managing outdated pull requests is time-consuming. Mergify's Merge Queue automates your pull request management & merging. It's fully integrated to GitHub & coordinated with any CI. Start focusing on code. Try Mergify for free.

  • rod

    A Devtools driver for web automation and scraping

    Project mention: Need help authenticating to Okta programatically. | /r/okta | 2023-07-03

    I have tried the following. 1. Login to Okta via browser programatically using go-rod. Which I managed to do so successfully, but I'm failing to load up Slack as it's stuck in the browser loader screen for Slack. 2. I tried to authenticate via Okta RESTful API. So far, I have managed to authenticate using {{domain}}/api/v1/authn, and then subsequently using MFA via the verify endpoint {{domain}}/api/v1/authn/factors/{{factorID}}/verify which returns me a sessionToken. From here, I can successfully create a sessionCookie which have proven quite useless to me. Perhaps I am doing it wrongly.

  • hakrawler

    Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

    Project mention: Find all src img on a public website? | /r/webdev | 2023-01-26

    This week i have used https://github.com/hakluke/hakrawler and I think it meets youre requirements.

  • cariddi

    Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

    Project mention: cariddi v1.3.1 is out🥳 | /r/opensource | 2023-03-24

    cariddi is an open source (https://github.com/edoardottt/cariddi) web security tool. It takes as input a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.

  • Dataflow kit

    Extract structured data from web sites. Web sites scraping.

  • webpalm

    WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.

    Project mention: 🌐 "WebPalm: Unleash Websites" 🌐 | /r/netsec | 2023-07-29
  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • antch

    Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  • bathyscaphe

    Fast, highly configurable, cloud native dark web crawler.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-08-23.

Go Crawling related posts


What are some of the best open-source Crawling projects in Go? This list will help you:

Project Stars
1 colly 20,763
2 Ferret 5,465
3 rod 4,182
4 hakrawler 3,941
5 cariddi 984
6 Dataflow kit 616
7 webpalm 285
8 antch 252
9 bathyscaphe 84
Collect and Analyze Billions of Data Points in Real Time
Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.