Python web-crawler

Open-source Python projects categorized as web-crawler

Top 7 Python web-crawler Projects

  • PSpider


  • kochat

    Opensource Korean chatbot framework

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • spidy Web Crawler

    The simple, easy to use command line web crawler.

  • Ignareo-ISML-auto-voter

    Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

  • GoodreadsScraper

    Scrape data from Goodreads using Scrapy and Selenium :books:

  • CobWeb-lnx

    CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

    Project mention: Quem já contribuiu e quem já usou projectos open-source? | /r/devpt | 2023-06-30
  • Python

    This repository contains the python source code, containing more than 40 python projects, involving many fields.仓库用于储存python源代码, 包含40多个python项目,涉及爬虫、算法、OpenGL、tkinter、面向对象编程等多个领域。 (by qfcy)

    Project mention: Potentially malicious package | /r/learnpython | 2023-12-05

    Recently, I tried to install the PyObject package, but accidentally ran pip install pyobject. The install failed with a Unicode error, but I am assuming that some code was run during the setup process. The main repo seems to be here. I'm just paranoid that it might be malicious, as there is a file called in the root of the project. I've looked over it and am of the opinion that it is most likely safe, but would just like a second opinion.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-12-05.

Python web-crawler related posts


What are some of the best open-source web-crawler projects in Python? This list will help you:

Project Stars
1 PSpider 1,811
2 kochat 442
3 spidy Web Crawler 322
4 Ignareo-ISML-auto-voter 189
5 GoodreadsScraper 115
6 CobWeb-lnx 38
7 Python 4
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives