Python web-crawler

Open-source Python projects categorized as web-crawler

Top 5 Python web-crawler Projects

  • PSpider


  • kochat

    Opensource Korean chatbot framework

  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code.

  • spidy Web Crawler

    The simple, easy to use command line web crawler.

  • Ignareo-ISML-auto-voter

    Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (2022 ISML is last ISML)

  • GoodreadsScraper

    Scrape data from Goodreads using Scrapy and Selenium :books:

    Project mention: [OC] Top 10 Fantasy Books of the 21st Century According to GoodReads | | 2022-05-20

    Used this code to get the data:

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-05-20.


What are some of the best open-source web-crawler projects in Python? This list will help you:

Project Stars
1 PSpider 1,727
2 kochat 335
3 spidy Web Crawler 297
4 Ignareo-ISML-auto-voter 181
5 GoodreadsScraper 68
Truly a developer’s best friend
Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.