Top 10 Go Crawling Projects

colly

39 22,120 6.0 Go

Elegant Scraper and Crawler Framework for Golang

Project mention: Scraping the full snippet from Google search result | dev.to | 2024-01-01

SerpApi focuses on scraping search results. That's why we need extra help to scrape individual sites. We'll use GoColly package.

Ferret

0 5,615 3.1 Go

Declarative web scraping
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
rod

20 4,750 7.9 Go

A Devtools driver for web automation and scraping

Project mention: Need help authenticating to Okta programatically. | /r/okta | 2023-07-03

I have tried the following. 1. Login to Okta via browser programatically using go-rod. Which I managed to do so successfully, but I'm failing to load up Slack as it's stuck in the browser loader screen for Slack. 2. I tried to authenticate via Okta RESTful API. So far, I have managed to authenticate using {{domain}}/api/v1/authn, and then subsequently using MFA via the verify endpoint {{domain}}/api/v1/authn/factors/{{factorID}}/verify which returns me a sessionToken. From here, I can successfully create a sessionCookie which have proven quite useless to me. Perhaps I am doing it wrongly.

hakrawler

3 4,225 0.0 Go

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
cariddi

7 1,348 6.7 Go

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
Dataflow kit

0 636 0.0 Go

Extract structured data from web sites. Web sites scraping.
webpalm

11 324 8.2 Go

🕸️ Crawl in the web network

Project mention: Modern automated data miner (scrapper) | news.ycombinator.com | 2024-02-08

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
antch

0 255 0.0 Go

Antch, a fast, powerful and extensible web crawling & scraping framework for Go
seonaut

3 121 8.9 Go

Open source SEO auditing tool.
bathyscaphe

1 92 5.6 Go

Fast, highly configurable, cloud native dark web crawler.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-08.