Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work. Learn more →
Top 22 Python Web Crawling Projects
-
Project mention: Is there a program available for bulk image reverse searching? | reddit.com/r/AskProgramming | 2023-02-02
In the past I used stuff like beautifulsoup for webscraping but I’ve heard good things about https://scrapy.org/
-
-
Sonar
Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
Project mention: 8 Most Popular Python HTML Web Scraping Packages with Benchmarks | dev.to | 2023-02-01
requests-html
-
-
Try with Mechanicalsoup https://mechanicalsoup.readthedocs.io/en/stable/
-
-
InfluxDB
Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.
-
-
-
feedparser
-
-
-
-
-
google-search-results is a SerpApi API package.
-
-
-
-
-
FastImage
Python library that finds the size / type of an image given its URI by fetching as little as needed (by bmuller)
-
-
Mariner
This a is mirror of Gitlab repository. Open your issues and pull requests there. (by radek-sprta)
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-02-02.
Python Web Crawling related posts
- Is there a program available for bulk image reverse searching?
- 8 Most Popular Python HTML Web Scraping Packages with Benchmarks
- How I used Scrapy for my ML Project
- scrapy.Request(url, callback) vs response.follow(url, callback)
- Newb learning GitHub & Python. Projects?
- Looking for a good open source web scraping tool
- Automatsko prebacivanje komentara sa neta u .txt fajl ili slicno
-
A note from our sponsor - Sonar
www.sonarsource.com | 4 Feb 2023
Index
What are some of the best open-source Web Crawling projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Scrapy | 46,031 |
| 2 | pyspider | 15,726 |
| 3 | requests-html | 12,923 |
| 4 | portia | 8,738 |
| 5 | MechanicalSoup | 4,306 |
| 6 | RoboBrowser | 3,647 |
| 7 | Grab | 2,259 |
| 8 | gain | 2,015 |
| 9 | PSpider | 1,735 |
| 10 | feedparser | 1,542 |
| 11 | cola | 1,459 |
| 12 | Sukhoi | 873 |
| 13 | MSpider | 344 |
| 14 | spidy Web Crawler | 304 |
| 15 | google-search-results-python | 263 |
| 16 | Crawley | 174 |
| 17 | brownant | 157 |
| 18 | Demiurge | 109 |
| 19 | Pomp | 61 |
| 20 | FastImage | 28 |
| 21 | microwler | 11 |
| 22 | Mariner | 2 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com