Go Scraper

Open-source Go projects categorized as Scraper

Top 23 Go Scraper Projects

  • lux

    👾 Fast and simple video download library and CLI tool written in Go

    Project mention: Bilibili download stalls at around 30-60% | /r/youtubedl | 2023-05-18

    Not a fix, but I tend to use lux when downloading from bilibili. It is faster too.

  • colly

    Elegant Scraper and Crawler Framework for Golang

    Project mention: New modern web crawling tool | news.ycombinator.com | 2023-04-30

    Sounds cool, but how is this different from Colly: https://github.com/gocolly/colly?

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • Ferret

    Declarative web scraping

  • rod

    A Devtools driver for web automation and scraping

    Project mention: Library to convert HTML to pdf in Golang | /r/golang | 2023-05-22
  • Geziyor

    Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  • till

    DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

  • mangal

    📖 The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!

    Project mention: What application handles manga downloads? | /r/selfhosted | 2023-05-19
  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • finance-go

    :bar_chart: Financial markets data library implemented in go.

    Project mention: finance-go: NEW Data - star count:602.0 | /r/algoprojects | 2023-05-13
  • Dataflow kit

    Extract structured data from web sites. Web sites scraping.

  • ant

    A web crawler for Go (by yields)

  • GMDB

    GMDB is the ultra-simple, cross-platform Movie Library with Features (Search, Take Note, Watch Later, Like, Import, Learn, Instantly Torrent Magnet Watch)

  • dorkscout

    DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

    Project mention: Automatizovani Google Dorking | /r/programiranje | 2023-04-14
  • demeter

    Demeter is a tool for scraping the calibre web ui

  • meteor

    Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog. (by odpf)

    Project mention: Modern open-source data platform that empowers organizations to discover, transform, analyse and secure data faster and efficiently. | /r/dataengineering | 2022-06-21

    Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.

  • scraply

    Scraply a simple dom scraper to fetch information from any html based website

    Project mention: scraply v3.0.0 - a very simple scraping too using CSS selectors and jQuery like functions | /r/golang | 2022-07-06
  • spidy

    Domain names collector - Crawl websites and collect domain names along with their availability status. (by twiny)

    Project mention: Share Your Code.. Share your most unique piece of Go code. | /r/golang | 2022-10-15

    1 - Expired domain scrapper => https://github.com/twiny/spidy 2 - A sample & efficient web crawler => https://github.com/twiny/wbot 3 - A mini blockchain scanner => https://github.com/twiny/blockscan 4 - A Snake Game => https://github.com/twiny/snaky

  • grab

    Configurable Scraper & Downloader, Powered by RegExp and Go (by everdrone)

    Project mention: Looking for approachable OSS project or mentor | /r/golang | 2022-09-15

    Hey, I am also relatively new to Go The project I’m maintaining is Grab

  • rrip

    Bulk image downloader for reddit.

    Project mention: Command line tool to bulk download images from reddit | /r/commandline | 2023-02-08
  • xdsl-exporter

    xDSL Prometheus Exporter

    Project mention: I created Prometheus Exporter with Go to scrape my xDSL Modem stats | /r/golang | 2023-03-17
  • fitter

    New way for collect information from the API's/Websites (by PxyUp)

    Project mention: Show HN: Library for scrape internet like GQL | news.ycombinator.com | 2023-03-24
  • go-recipe

    Go package for scraping website recipes

  • goskyr

    A configurable command-line web scraper written in go with auto configuration capability

    Project mention: No code command line webscraper | /r/webscraping | 2023-03-09

    I am currently building a webscraper, called goskyr, that can be run from the command line and is supposed to be easily configurable. So instead of having to write code to scrape a website you'd just write a configuration snippet and run the scraper. I realize that there are a number of gui based scraping services that make it extremely easy to setup a scraping process for any website, so for people having no coding experience whatsoever that would probably be the easiest solution. I'm trying to come close to those gui based solutions in terms of functionality by providing a 'smart' way of finding potentially interesting data/fields and letting the user select a subset in a terminal based ui. Also date extraction & parsing and the newly added machine learning capability is probably worth mentioning. Still, those other, gui based solutions are really awesome, eg octoparse or scrapestorm.

  • moviestills

    A small CLI app to scrap high-quality movie snapshots from various websites.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-05-22.

Go Scraper related posts

Index

What are some of the best open-source Scraper projects in Go? This list will help you:

Project Stars
1 lux 21,142
2 colly 19,649
3 Ferret 5,366
4 rod 3,836
5 Geziyor 1,932
6 till 799
7 mangal 685
8 finance-go 601
9 Dataflow kit 593
10 ant 273
11 GMDB 226
12 dorkscout 199
13 demeter 163
14 meteor 145
15 scraply 123
16 spidy 116
17 grab 63
18 rrip 56
19 xdsl-exporter 44
20 fitter 40
21 go-recipe 20
22 goskyr 19
23 moviestills 13
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com