Python scrapyd

Open-source Python projects categorized as scrapyd
Edit details
Spider Scrapy scrapyd-ui Dashboard scrapy-visualization
Save 47% on cloud hosting with autoscaling that just works
Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.
judoscale.com
featured

Top 5 Python scrapyd Projects

scrapyd

  1. Gerapy

    Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

  2. Judoscale

    Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

    Judoscale logo

  3. scrapydweb

    Web app for Scrapyd cluster management, Scrapy log analysis & visualization, Auto packaging, Timer tasks, Monitor & Alert, and Mobile UI. Docs 文档 :point_right:

  4. SpiderKeeper

    admin ui for scrapy/open source scrapinghub

  5. estela

    estela, an elastic web scraping cluster 🕸

  6. scrapeops-scrapy-sdk

    Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.

  7. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python scrapyd discussion

Log in or Post with

Python scrapyd related posts

  • Distribution of gross and net salaries on r/BESalary [OC]

    1 project | /r/BESalary | 1 Jul 2023

  • Best scrapydweb fork

    1 project | /r/scrapy | 16 Nov 2022

  • Free Python Scrapy 5-Part Mini Course

    1 project | /r/scrapy | 20 Oct 2022

  • How do you guys manage a large amount of scrapers?

    1 project | /r/webscraping | 19 Sep 2022

  • Where to buy multiple proxy server access?

    1 project | /r/webdev | 11 Sep 2022

  • The Python Scrapy Playbook

    3 projects | /r/learnpython | 7 Jun 2022

  • Free tool to monitor Scrapy spiders

    1 project | news.ycombinator.com | 3 Jun 2022
  • A note from our sponsor - InfluxDB
    influxdata.com | 21 Apr 2025
    Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems. Learn more →

Index

What are some of the best open-source scrapyd projects in Python? This list will help you:

# Project Stars
1 Gerapy 3,424
2 scrapydweb 3,265
3 SpiderKeeper 2,749
4 estela 180
5 scrapeops-scrapy-sdk 36

Sponsored
Save 47% on cloud hosting with autoscaling that just works
Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.
judoscale.com
Loading...