Top 23 Python Scrapy Projects

Scrapy

• Add a project

• scrapy-redis

  4 5,543 6.0 Python

  Redis-based components for Scrapy.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• Gerapy

  1 3,359 5.4 Python

  Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

  

• scrapydweb

  6 3,173 2.6 Python

  Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. DEMO :point_right:

• scrapy-splash

  3 3,153 0.0 Python

  Scrapy+Splash for JavaScript integration

  

• SpiderKeeper

  1 2,738 0.0 Python

  admin ui for scrapy/open source scrapinghub

• advertools

  1 1,159 8.9 Python

  advertools - online marketing productivity and analysis tools

• scrapy-playwright

  14 1,035 8.1 Python

  🎭 Playwright integration for Scrapy

  

  Project mention: Current problems and mistakes of web scraping in Python and tricks to solve them! | dev.to | 2024-08-22

  Middleware libraries are written by the community and are extending their functionality. For example, scrapy-playwright.

• scrapyrt

  3 837 5.5 Python

  HTTP API for Scrapy spiders

  

• scrapy-rotating-proxies

  4 738 0.0 Python

  use multiple proxies with Scrapy

  

• scrapy-fake-useragent

  3 686 2.3 Python

  Random User-Agent middleware based on fake-useragent

• alltheplaces

  8 636 10.0 Python

  A set of spiders and scrapers to extract location information from places that post their location on the internet.

  

  Project mention: AllThePlaces.xyz | news.ycombinator.com | 2024-08-19

  An open web data scraping dataset of CC 0 licenced POI, written in python with the scrapy framework.

  https://github.com/alltheplaces/alltheplaces

• estela

  10 174 5.1 Python

  estela, an elastic web scraping cluster 🕸

  

• GoodreadsScraper

  1 129 2.7 Python

  Scrape data from Goodreads using Scrapy and Selenium :books:

• scrapy-cloudflare-middleware

  1 105 0.0 Python

  A Scrapy middleware to bypass the CloudFlare's anti-bot protection

• scrapy-crawl-once

  1 79 0.0 Python

  Scrapy middleware which allows to crawl only new content

  

• open-gov-crawlers

  13 66 6.8 Python

  Parse government documents into well formed JSON

  

• scrapy-mysql-pipeline

  1 49 10.0 Python

  scrapy mysql pipeline

• scrapeops-scrapy-sdk

  11 37 3.9 Python

  Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

  

• scrapingant-client-python

  1 36 5.0 Python

  ScrapingAnt API client for Python.

  

• burplist

  5 13 6.7 Python

  Web crawler for Burplist, a search engine for craft beers in Singapore

• hltv-scraping

  1 10 0.0 Python

  Scraping data from hltv.org

• nse-stock-scraper

  1 10 5.3 Python

  This is Web Scraper utilizing Scrapy Framework, MongoDB and AfricasTalking to get stock prices for companies listed on the Nairobi Stock Exchange. This project will store ticker name and price as well notify via SMS once properly setup via AfricasTalking.

• scrapy-folder-tree

  2 9 6.0 Python

  A scrapy pipeline which stores files using folder trees.

Python Scrapy related posts

• AllThePlaces.xyz

  2 projects | news.ycombinator.com | 19 Aug 2024

• Announcing Crawlee Python: Now you can use Python to build reliable web crawlers

  4 projects | dev.to | 9 Jul 2024

• Web Scraping Dynamic Websites With Scrapy Playwright

  1 project | dev.to | 6 Mar 2024

• Differentiating between hypermarkets and supermarkets.

  1 project | /r/openstreetmap | 9 Dec 2023

• Meta, Microsoft and Amazon team up on maps project

  1 project | news.ycombinator.com | 26 Jul 2023

• Distribution of gross and net salaries on r/BESalary [OC]

  1 project | /r/BESalary | 1 Jul 2023

• How to make scrapy run multiple times on the same URLs?

  2 projects | /r/scrapy | 26 Jun 2023
• A note from our sponsor - SaaSHub
  www.saashub.com | 5 Dec 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!