The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 Python Scrapy Projects
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
scrapydweb
Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
alltheplaces
A set of spiders and scrapers to extract location information from places that post their location on the internet.
-
-
-
-
scrapeops-scrapy-sdk
Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
-
-
nse-stock-scraper
This is Web Scraper utilizing Scrapy Framework, MongoDB and AfricasTalking to get stock prices for companies listed on the Nairobi Stock Exchange. This project will store ticker name and price as well notify via SMS once properly setup via AfricasTalking.
-
NSFW_Scraper
Scraper to get Meta-data of all available scenes and movies and storing it to Postgresql every few days.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
scrapy-playwright is an integration between Scrapy and Playwright. It enables scraping dynamic web pages with Scrapy by processing the web scraping requests using a Playwright instance.
Project mention: Differentiating between hypermarkets and supermarkets. | /r/openstreetmap | 2023-12-09Maybe a different approach? https://www.alltheplaces.xyz/ has stores grouped by name
Project mention: Distribution of gross and net salaries on r/BESalary [OC] | /r/BESalary | 2023-07-01My favourite scrapingtool is Scrappy, requires some Python knowledge but there are some very good tutorials about it on https://scrapeops.io
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Scrapy related posts
- Web Scraping Dynamic Websites With Scrapy Playwright
- Differentiating between hypermarkets and supermarkets.
- Meta, Microsoft and Amazon team up on maps project
- Distribution of gross and net salaries on r/BESalary [OC]
- How to make scrapy run multiple times on the same URLs?
- How do you handle CAPTCHA pages appearing in some of the rotating proxies you use?
- Scrapy & splash guide
-
A note from our sponsor - WorkOS
workos.com | 25 Apr 2024
Index
What are some of the best open-source Scrapy projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | scrapy-redis | 5,451 |
| 2 | Gerapy | 3,210 |
| 3 | scrapy-splash | 3,051 |
| 4 | scrapydweb | 3,001 |
| 5 | SpiderKeeper | 2,704 |
| 6 | advertools | 1,055 |
| 7 | scrapy-playwright | 828 |
| 8 | scrapyrt | 816 |
| 9 | scrapy-rotating-proxies | 705 |
| 10 | scrapy-fake-useragent | 681 |
| 11 | alltheplaces | 528 |
| 12 | estela | 153 |
| 13 | GoodreadsScraper | 115 |
| 14 | scrapy-cloudflare-middleware | 102 |
| 15 | scrapy-crawl-once | 77 |
| 16 | open-gov-crawlers | 61 |
| 17 | scrapy-mysql-pipeline | 48 |
| 18 | scrapeops-scrapy-sdk | 36 |
| 19 | scrapingant-client-python | 31 |
| 20 | burplist | 11 |
| 21 | hltv-scraping | 10 |
| 22 | nse-stock-scraper | 10 |
| 23 | NSFW_Scraper | 8 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com