Top 9 Go web-scraping Projects

rod

20 4,727 7.9 Go

A Devtools driver for web automation and scraping

Project mention: Need help authenticating to Okta programatically. | /r/okta | 2023-07-03

I have tried the following. 1. Login to Okta via browser programatically using go-rod. Which I managed to do so successfully, but I'm failing to load up Slack as it's stuck in the browser loader screen for Slack. 2. I tried to authenticate via Okta RESTful API. So far, I have managed to authenticate using {{domain}}/api/v1/authn, and then subsequently using MFA via the verify endpoint {{domain}}/api/v1/authn/factors/{{factorID}}/verify which returns me a sessionToken. From here, I can successfully create a sessionCookie which have proven quite useless to me. Perhaps I am doing it wrongly.
till

5 807 1.8 Go

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
InfluxDB

www.influxdata.com
sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
google-maps-scraper

5 583 7.1 Go

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place (by gosom)

Project mention: Show HN: A Google Maps Scraper | news.ycombinator.com | 2023-12-03
crawley

8 225 6.7 Go

The unix-way web crawler (by s0rg)
JsonGenius

12 150 6.1 Go

Get structured JSON data from any page.

Project mention: Show HN: SingleAPI – Convert the Internet into your own API | news.ycombinator.com | 2023-10-17

isn’t this just using jsongenius[1]
[1] https://github.com/semanser/JsonGenius
cascadia

1 134 4.7 Go

Go cascadia package command line CSS selector
scrapemate

1 55 6.7 Go

Golang Crawling and scraping framework (by gosom)
WorkOS

workos.com
sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
yast

1 28 1.7 Go

Yet Another Streaming Tool

Project mention: [OpenSource] I am building high performance Plex alternative in Go for Movies and TV Show | /r/golang | 2023-06-02

I also build a similar tool, it let's you choose and play movies. I used webtorrent behind the scenes. https://github.com/qascade/yast
snagtag

1 5 8.7 Go

Scrape social media platforms such as Youtube and Tiktok, powered by Go.

Project mention: Snagtag - Social Media Scraping Tool | /r/golang | 2023-10-26

Link: SnagTag

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-12-03.