SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Go Crawler Projects
👾 Fast and simple video download library and CLI tool written in GoProject mention: Bilibili download stalls at around 30-60% | /r/youtubedl | 2023-05-18
Not a fix, but I tend to use lux when downloading from bilibili. It is faster too.
Elegant Scraper and Crawler Framework for GolangProject mention: New modern web crawling tool | news.ycombinator.com | 2023-04-30
Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架Project mention: Self-hosted web scraper? | /r/selfhosted | 2023-01-03
Haven't tried but this project https://github.com/crawlab-team/crawlab looks promising.
Pholcus is a distributed high-concurrency crawler software written in pure golang
A next-generation crawling and spidering framework.Project mention: Originally a Covid project. Now a discount search engine. | /r/nextjs | 2023-02-05
Using a few different methods. Pulling the sites I'm using Puppeteer and Katana (https://github.com/projectdiscovery/katana). To process and extract the information is tricky, most websites selling things put time into their metadata; this does make it easier. Additionally, a lot of the larger stores have common patterns between them. Failing all of this, I trained a Tensor flow model to understand how to read product pages. However, it's far from perfect and a journey of continual improvement.
Declarative web scraping
Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and moreProject mention: cariddi v1.3.1 is out🥳 | /r/opensource | 2023-03-24
cariddi is an open source (https://github.com/edoardottt/cariddi) web security tool. It takes as input a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more.
The fastest dork scanner written in Go.
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
Antch, a fast, powerful and extensible web crawling & scraping framework for Go
DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targetsProject mention: Automatizovani Google Dorking | /r/programiranje | 2023-04-14
Rapid Smart Contract CrawlerProject mention: Chain Walker - Smart Contract (RCP/IPC) Crawler 👻🧛♂️ | /r/netsec | 2022-06-18
The unix-way web crawler (by s0rg)Project mention: github/crawley v1.5.0 released | /r/golang | 2022-10-08
crawley project: https://github.com/s0rg/crawley
Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)Project mention: Share Your Code.. Share your most unique piece of Go code. | /r/golang | 2022-10-15
1 - Expired domain scrapper => https://github.com/twiny/spidy 2 - A sample & efficient web crawler => https://github.com/twiny/wbot 3 - A mini blockchain scanner => https://github.com/twiny/blockscan 4 - A Snake Game => https://github.com/twiny/snaky
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
rotating open proxy multiplexerProject mention: SLRP – rotating open proxy multiplexer | news.ycombinator.com | 2022-07-12
Fast, highly configurable, cloud native dark web crawler.
Fast website scraper and wordlist generator
Google Search Results GoLang API
Open source SEO auditing tool.
WebPalm is a powerful command-line tool for website mapping and web scraping. With its recursive approach, it can generate a complete tree of all webpages and their links on a website. It can also extract data from the body of each page using regular expressions, making it an ideal tool for web scraping and data extraction.Project mention: webpalm | /r/redteamsec | 2023-06-05
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Go Crawler related posts
1 project | /r/redteamsec | 5 Jun 2023
Bilibili download stalls at around 30-60%
1 project | /r/youtubedl | 18 May 2023
1 project | /r/bugbounty | 15 May 2023
New Modern Crawling tool written with go
1 project | news.ycombinator.com | 3 May 2023
New Modern Fast Crawler
1 project | news.ycombinator.com | 2 May 2023
Webpalm - Modern fast web crawling tool in go
1 project | /r/CKsTechNews | 1 May 2023
Modern fast web crawling tool in go
1 project | news.ycombinator.com | 1 May 2023
A note from our sponsor - #<SponsorshipServiceOld:0x00007f092083ad30>
www.saashub.com | 10 Jun 2023
What are some of the best open-source Crawler projects in Go? This list will help you: