Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work. Learn more →
Top 22 Python Web Crawling Projects
-
Project mention: Is there a program available for bulk image reverse searching? | reddit.com/r/AskProgramming | 2023-02-02
In the past I used stuff like beautifulsoup for webscraping but I’ve heard good things about https://scrapy.org/
-
-
Sonar
Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.
-
Project mention: 8 Most Popular Python HTML Web Scraping Packages with Benchmarks | dev.to | 2023-02-01
requests-html
-
-
Try with Mechanicalsoup https://mechanicalsoup.readthedocs.io/en/stable/
-
-
InfluxDB
Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.
-
-
-
feedparser
-
-
-
-
-
google-search-results is a SerpApi API package.
-
-
-
-
-
FastImage
Python library that finds the size / type of an image given its URI by fetching as little as needed (by bmuller)
-
-
Mariner
This a is mirror of Gitlab repository. Open your issues and pull requests there. (by radek-sprta)
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Python Web Crawling related posts
- Is there a program available for bulk image reverse searching?
- 8 Most Popular Python HTML Web Scraping Packages with Benchmarks
- How I used Scrapy for my ML Project
- scrapy.Request(url, callback) vs response.follow(url, callback)
- Newb learning GitHub & Python. Projects?
- Looking for a good open source web scraping tool
- Automatsko prebacivanje komentara sa neta u .txt fajl ili slicno
-
A note from our sponsor - Sonar
www.sonarsource.com | 4 Feb 2023
Index
What are some of the best open-source Web Crawling projects in Python? This list will help you:
Project | Stars | |
---|---|---|
1 | Scrapy | 46,031 |
2 | pyspider | 15,726 |
3 | requests-html | 12,923 |
4 | portia | 8,738 |
5 | MechanicalSoup | 4,306 |
6 | RoboBrowser | 3,647 |
7 | Grab | 2,259 |
8 | gain | 2,015 |
9 | PSpider | 1,735 |
10 | feedparser | 1,542 |
11 | cola | 1,459 |
12 | Sukhoi | 873 |
13 | MSpider | 344 |
14 | spidy Web Crawler | 304 |
15 | google-search-results-python | 263 |
16 | Crawley | 174 |
17 | brownant | 157 |
18 | Demiurge | 109 |
19 | Pomp | 61 |
20 | FastImage | 28 |
21 | microwler | 11 |
22 | Mariner | 2 |