Top 23 Python Crawler Projects

• Scrapy

  76 45,702 9.6 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

  Project mention: How I used Scrapy for my ML Project | dev.to | 2023-01-27

  I wanted to invest my time and energy in learning the fastest, most efficient one, that can scale with my as my projects get more and more complex scrapy. After all, I want my projects to shine so bright in my cv it blinds the recruiter's eyes.

• pyspider

  0 15,713 0.0 Python

  A Powerful Spider(Web Crawler) System in Python.

• Sonar

  www.sonarsource.com

  sponsored

  Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  

• newspaper

  5 12,386 0.0 Python

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

  Project mention: Gathering News Headlines | reddit.com/r/Automate | 2022-08-08

• Photon

  0 9,334 1.3 Python

  Incredibly fast crawler designed for OSINT. (by s0md3v)

• scrapy-redis

  2 5,227 7.3 Python

  Redis-based components for Scrapy.

  Project mention: Ask HN: What are the best tools for web scraping in 2022? | news.ycombinator.com | 2022-08-10

  11. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis.

• autoscraper

  3 4,873 3.5 Python

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  Project mention: A Smart, Automatic, Fast and Lightweight Web Scraper for Python | reddit.com/r/webdev | 2022-12-02

• toapi

  0 3,356 0.0 Python

  Every web site provides APIs.

• InfluxDB

  www.influxdata.com

  sponsored

  Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.

  

• Grab

  0 2,257 6.2 Python

  Web Scraping Framework

• weibo-crawler

  1 2,159 6.2 Python

  新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

  Project mention: 紧急求助🆘备份@我能抱起120斤 | reddit.com/r/DoubanGoosegroup | 2022-07-15

• TorBot

  1 1,805 7.1 Python

  Dark Web OSINT Tool

  

  Project mention: List of resources | reddit.com/r/HackForUkraine | 2022-02-27

• PSpider

  0 1,732 3.9 Python

  简单易用的Python爬虫框架，QQ交流群：597510560

• news-please

  1 1,489 10.0 Python

  news-please - an integrated web crawler and information extractor for news that just works

  Project mention: Data extraction from news media outlets? | reddit.com/r/datasets | 2022-10-13

  Look at news-please, you can find it in GitHub. I did something similar and it was very helpful. You can hit me up with if you have any questions.

• OpenWPM

  0 1,248 5.2 Python

  A web privacy measurement framework

  

• grab-site

  10 959 7.3 Python

  The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  

  Project mention: How are you archiving websites you visit? | reddit.com/r/selfhosted | 2023-01-07

  After a lot of searching for a similar topic, this is a tool I found which works pretty well: https://github.com/ArchiveTeam/grab-site

• mlscraper

  3 815 8.7 Python

  🤖 Scrape data from HTML websites automatically by just providing examples

  Project mention: Pre-trained Webscraping Models | reddit.com/r/webscraping | 2022-12-04

• XSRFProbe

  0 791 0.0 Python

  The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

• scrapyrt

  1 758 0.7 Python

  HTTP API for Scrapy spiders

  

  Project mention: New to python and scrapy stuff but need this project to work so that I can do my data research and stuff easily in the future. | reddit.com/r/scrapy | 2022-04-19

• trafilatura

  6 730 8.2 Python

  Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

  Project mention: Testing fast installation in tear-down environment | reddit.com/r/learnpython | 2022-07-06

  I want to test how easy it is to install a package plus special extra dependencies to run a certain script in that package: https://github.com/adbar/trafilatura

• bookcorpus

  1 594 0.0 Python

  Crawl BookCorpus

  Project mention: Can chat GPT overtake Google if they play their cards right? | reddit.com/r/Futurology | 2022-12-23

• dotcommon

  1 591 0.0 Python

  What do people have in their dotfiles?

  Project mention: Dotcommon – common aliases and plugins on most updated GitHub repositories | news.ycombinator.com | 2022-08-13

• google-play-scraper

  2 490 6.9 Python

  Google play scraper for Python inspired by <facundoolano/google-play-scraper> (by JoMingyu)

  Project mention: Report: Analysis of 2.9 millions apps on Google Play | reddit.com/r/androiddev | 2022-11-08

  Its easy. python library: google-play-scraper.

• freshonions-torscraper

  0 439 0.7 Python

  Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

• Moodle-Downloader-2

  0 309 6.7 Python

  A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)

• SaaSHub

  www.saashub.com

  sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Crawler related posts

• I need data from a website. It is viable to create an API that scrapes the website and returns the data on an endpoint?
  2 projects | reddit.com/r/dotnet | 20 Dec 2022
• Can chat GPT overtake Google if they play their cards right?
  2 projects | reddit.com/r/Futurology | 23 Dec 2022
• A Smart, Automatic, Fast and Lightweight Web Scraper for Python
  1 project | reddit.com/r/webdev | 2 Dec 2022
• A next-gen crawling and spidering framework
  3 projects | news.ycombinator.com | 8 Nov 2022
• Report: Analysis of 2.9 millions apps on Google Play
  1 project | reddit.com/r/androiddev | 8 Nov 2022
• What else could be built using the LBRY protocol?
  5 projects | reddit.com/r/lbry | 30 Sep 2022
• Dotcommon – common aliases and plugins on most updated GitHub repositories
  1 project | news.ycombinator.com | 13 Aug 2022
• A note from our sponsor - InfluxDB
  www.influxdata.com | 28 Jan 2023

  InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!