|9 days ago||20 days ago|
|BSD 3-clause "New" or "Revised" License||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
13 ways to scrape any public data from any website
6 projects | dev.to | 7 Oct 2022
variable.css(".X5PpBb::text").get() # returns a text value variable.css(".gs_a").xpath("normalize-space()").get() # https://github.com/scrapy/parsel/issues/192#issuecomment-1042301716 variable.css(".gSGphe img::attr(srcset)").get() # returns a attribute value variable.css(".I9Jtec::text").getall() # returns a list of strings values variable.xpath('th/text()').get() # returns text value using xpath
Web Scraping With Python (An Ultimate Guide)
3 projects | reddit.com/r/Python | 15 Sep 2022
Something I don't see discussed when this topic is brought up is that Scrapy's HTML parsing library, parsel, can be installed separately from scrapy itself. You can use it in place of beautifulsoup and, imo, it's much easier to use.
How to Crawl the Web with Scrapy
7 projects | news.ycombinator.com | 13 Sep 2021
Learn how to scrape Trustpilot reviews using Go
4 projects | dev.to | 4 Feb 2023
github.com/gocolly/colly - popular and widely-used library for web scraping in Go. It provides a higher-level API than net/http and makes it easier to extract information from websites. It also provides features such as concurrency, automatic request retries, and support for cookies and sessions.
Go for web scraping
5 projects | reddit.com/r/golang | 18 Nov 2022
http://go-colly.org/ is excellent
FAQs on my side project
2 projects | reddit.com/r/SideProject | 24 Oct 2022
I wrote a script in Go to scrape jobs from these companies. If you are interested in writing scraper with Go, check out http://go-colly.org/ a potent tool for scraping in Go. I scrape every day just to keep up to date with the jobs.
Dan terjadi lagi
3 projects | reddit.com/r/indonesia | 16 Oct 2022
help me migrate from nodejs to go
3 projects | reddit.com/r/golang | 9 Oct 2022
Most Popular GoLang Frameworks
13 projects | dev.to | 20 Sep 2022
Is it possible to scrape a website protected by Cloudflare?
6 projects | reddit.com/r/golang | 10 Sep 2022
Preferred Programming Languages for Web Scraping
2 projects | reddit.com/r/programming | 24 Aug 2022
in addition the article mention that "Also, C++ is not good for creating web crawlers, is programming language is great for simple web scraping" but in reality when you need a fast and performant crawler C++ will be beat languages like ruby and python from performance perspective , the same for golang programming language which will perform better most of the programming languages written on the article and has bunch of very good libraries like http://go-colly.org/ in addition to the https://pkg.go.dev/net/http which comes part of the core
Scraping websites with go
2 projects | reddit.com/r/golang | 21 Aug 2022
Strange that the author didn’t mention the most popular and, frankly, convenient way: Colly.
Is there a library similar to HTMLUnit in GO?
5 projects | reddit.com/r/golang | 19 Aug 2022
What are some alternatives?
GoQuery - A little like that j-thing, only in Go.
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
xpath - XPath package for Golang, supports HTML, XML, JSON document query.
rod - A Devtools driver for web automation and scraping
Ferret - Declarative web scraping
Geziyor - Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Pholcus - Pholcus is a distributed high-concurrency crawler software written in pure golang
google-search-results-golang - Google Search Results GoLang API
Playwright - Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
Dataflow kit - Extract structured data from web sites. Web sites scraping.
jsonrpconion - Library for building JSON RPC services on Tor network
gichidan - Gichidan - CLI wrapper for Ichidan deep-web search engine.