Top 22 Python Web Crawling Projects

• Scrapy

  67 42,622 8.9 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

  Project mention: Legalität von Web scraping | reddit.com/r/de_EDV | 2022-01-22

• pyspider

  0 15,288 0.0 Python

  A Powerful Spider(Web Crawler) System in Python.

• Scout APM

  scoutapm.com

  sponsored

  Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  

• requests-html

  6 12,324 0.0 Python

  Pythonic HTML Parsing for Humans™

  

  Project mention: How to make all https traffic in program go through a specific proxy? | reddit.com/r/learnpython | 2021-12-24

• portia

  0 8,309 0.0 Python

  Visual scraping for Scrapy

  

• MechanicalSoup

  0 3,884 3.7 Python

  A Python library for automating interaction with websites.

  

• RoboBrowser

  0 3,574 0.0 Python

• Grab

  0 2,154 0.0 Python

  Web Scraping Framework

• SonarLint

  www.sonarlint.org

  sponsored

  Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  

• gain

  0 2,005 0.0 Python

  Web crawling framework based on asyncio.

• PSpider

  0 1,619 4.7 Python

  简单易用的Python爬虫框架，QQ交流群：597510560

• cola

  0 1,423 0.0 Python

  A high-level distributed crawling framework.

• feedparser

  0 1,352 6.6 Python

  Parse feeds in Python

• Sukhoi

  0 881 0.0 Python

  Minimalist and powerful Web Crawler.

  

• MSpider

  0 344 0.0 Python

  Spider

• spidy Web Crawler

  0 277 3.4 Python

  The simple, easy to use command line web crawler.

• google-search-results-python

  0 180 6.2 Python

  Google Search Results via SERP API pip Python Package

  

• Crawley

  0 172 0.0 Python

  Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.

• brownant

  0 157 0.0 Python

  Brownant is a web data extracting framework.

  

• Demiurge

  0 107 0.0 Python

  PyQuery-based scraping micro-framework.

• Pomp

  0 61 0.0 Python

  Screen scraping and web crawling framework

• FastImage

  0 28 0.0 Python

  Python library that finds the size / type of an image given its URI by fetching as little as needed (by bmuller)

• microwler

  0 9 5.0 Python

  A micro-framework for asynchronous deep crawls and web scraping with Python

  

• Mariner

  0 2 2.2 Python

  This a is mirror of Gitlab repository. Open your issues and pull requests there. (by radek-sprta)

Python Web Crawling related posts

• Legalität von Web scraping
  1 project | reddit.com/r/de_EDV | 22 Jan 2022
• Feedback Request: Utilities for web scraping
  2 projects | reddit.com/r/Python | 30 Dec 2021
• Top 13 Web scraping tools in 2022
  1 project | reddit.com/r/u_digitally_rajat | 28 Dec 2021
• How is ArchiveBox?
  4 projects | reddit.com/r/selfhosted | 27 Dec 2021
• How to make all https traffic in program go through a specific proxy?
  1 project | reddit.com/r/learnpython | 24 Dec 2021
• Do research web crawler programs exist?
  1 project | reddit.com/r/AskProgramming | 17 Dec 2021
• Web scraping data with pagination?
  1 project | reddit.com/r/learnpython | 2 Dec 2021

Index

Sponsored

OPS - Build and Run Open Source Unikernels

Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

github.com/nanovms

Get the trending Python projects
with our weekly report!
» Subscribe «