Top 23 Python Crawler Projects

• Scrapy

  67 42,525 9.0 Python

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  

  Project mention: Konohagakure Search | dev.to | 2022-01-09

  Scrapy

• pyspider

  0 15,274 0.0 Python

  A Powerful Spider(Web Crawler) System in Python.

• SonarLint

  www.sonarlint.org

  sponsored

  Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  

• newspaper

  8 11,599 0.0 Python

  News, full-text, and article metadata extraction in Python 3. Advanced docs:

  Project mention: Is there a web text extraction library for reader mode written in Java/Kotlin? | reddit.com/r/webdev | 2021-12-20

  I have searched the web but the library I have found was for Python only. I need a library written in Java or Kotlin so that I could use it on Android. Is there any library for that? If you know that there is no such Java library, please let me know that so that I could stop searching.

• Photon

  3 8,388 0.0 Python

  Incredibly fast crawler designed for OSINT. (by s0md3v)

  Project mention: How is ArchiveBox? | reddit.com/r/selfhosted | 2021-12-27

  If you need more advanced recursive spider/crawling ability beyond --depth=1, check out Browsertrix, Photon, or Scrapy and pipe the outputted URLs into ArchiveBox.

• autoscraper

  1 4,184 0.4 Python

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  Project mention: Turn Any Website Into An API with AutoScraper and FastAPI | dev.to | 2021-04-24

  In this article, we will learn how to create a simple e-commerce search API with multiple platform support: eBay and Amazon. AutoScraper and FastAPi provide the ability to create a powerful JSON API for the date. With Playwright's help, we'll extend our scraper and avoid blocking by using ScrapingAnt's web scraping API.

• toapi

  0 3,216 2.3 Python

  Every web site provides APIs.

• instagram-scraper

  2 2,239 0.8 Python

  scrapes medias, likes, followers, tags and all metadata. Inspired by instagram-php-scraper,bot (by realsirjoe)

  Project mention: Help in scraping Instagram | reddit.com/r/webscraping | 2021-10-09

  Yes: https://github.com/realsirjoe/instagram-scraper

• OPS

  github.com/nanovms

  sponsored

  OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  

• PSpider

  0 1,618 3.0 Python

  简单易用的Python爬虫框架，QQ交流群：597510560

• grab-site

  12 803 6.2 Python

  The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  

  Project mention: is there something like DownThemAll! that can be run without installing anything? | reddit.com/r/Archiveteam | 2022-01-16

• XSRFProbe

  1 722 0.0 Python

  The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

  Project mention: Write-Up: TryHackMe Web Fundamentals - ZTH: Obscure Web Vulns | dev.to | 2022-01-05

  This section does not have an actual challenge, but it requires you to get familiar with the xsrfprobe library. You could either install the library to your device and use xsrfprobe --help to find the right argument for generating a POC, or use the official documentation from the web. Hint: Find out which command would let you craft an actual, malicious request.

• bookcorpus

  1 516 1.1 Python

  Crawl BookCorpus

  Project mention: On the Danger of Stochastic Parrots [pdf] | news.ycombinator.com | 2021-03-01

  The GPT-3 paper (section 2.2) mentions using two datasets referred to as "books1" and "books2", which are 12B and 55B byte pair encoded tokens each.

  Project Gutenberg has 3B word tokens I believe, so it seems like it could be one of them, assuming the ratio of word tokens to byte-pair tokens is something like 3:12 to 3:55.

  Another likely candidate alongside Gutenberg is libgen, apparently, and looks like there have been successful efforts to create a similar dataset called bookcorpus: https://github.com/soskek/bookcorpus/issues/27). The discussion on that github issue suggests bookcorpus is very similar to "books2", which would make gutenberg "books1"?

  This might be why the paper is intentionally vague about the books used?

• dotcommon

  0 509 0.0 Python

  What do people have in their dotfiles?

• freshonions-torscraper

  2 392 0.2 Python

  Fresh Onions is an open source TOR spider / hidden service onion crawler hosted at zlal32teyptf4tvi.onion

  Project mention: How do I explore sites that aren't in lists? | reddit.com/r/TOR | 2021-08-11

• trafilatura

  2 359 9.3 Python

  Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)

  Project mention: Most universal way to grab relevant webpage content | reddit.com/r/webscraping | 2021-11-30

  Check out this python package https://github.com/adbar/trafilatura it seems to do what you are looking for. You may need to use an automated browser like selenium to first load the content of the page if you want it to work on all sites, even those that use js

• mlscraper

  2 353 0.0 Python

  🤖 Scrape data from HTML websites automatically with Machine Learning

  Project mention: mlscraper: Scrape data from HTML pages automatically with Machine Learning | news.ycombinator.com | 2021-07-05

• google-play-scraper

  1 284 5.8 Python

  Google play scraper for Python inspired by <facundoolano/google-play-scraper> (by JoMingyu)

  Project mention: Some CN players are conducting data analysis on Genshin’s reviews on Google Play | reddit.com/r/Genshin_Impact | 2021-09-30

• spidy Web Crawler

  0 276 3.4 Python

  The simple, easy to use command line web crawler.

• Moodle-Downloader-2

  1 210 9.1 Python

  A Moodle downloader that downloads course content fast from Moodle (eg. lecture pdfs)

  Project mention: Losing participation in an ONLINE ONLY CLASS. Why do professors act like their class is the only one? I expected this behavior in undergrad but not in grad school... | reddit.com/r/GradSchool | 2021-03-01

  That sounds terrible. From reading what you’ve written, the following may not be something you can do on your own, but maybe you have a technically inclined friend that can help you with it. There is a Linux based script that can be configured to automatically log on to Moodle and check it at specific intervals (and download the content). I suspect that running this script twice a day will help you ace those stats. https://github.com/C0D3D3V/Moodle-Downloader-2/issues

• bitextor

  2 199 9.4 Python

  Bitextor generates translation memories from multilingual websites

  

  Project mention: Bitextor: a tool for generating translation memories from multilingual websites | reddit.com/r/LanguageTechnology | 2021-04-20

• TorCrawl.py

  0 82 0.5 Python

  Crawl and extract (regular or onion) webpages through TOR network

• Python Testing Crawler

  0 68 2.8 Python

  A crawler for automated functional testing of a web application

  

• nebula-crawler

  7 65 9.5 Python

  🌌 A libp2p DHT crawler, monitor, and measurement tool that exposes timely information about DHT networks.

  Project mention: Nebula - A libp2p DHT crawler. | reddit.com/r/libp2p | 2021-07-17

• deepweb-scappering

  1 63 0.4 Python

  Discover hidden deepweb pages

  Project mention: DeepWeb Scapper: Descubre las páginas ocultas de #deepweb 👀 | reddit.com/r/u_esgeeks | 2021-04-07

Python Crawler related posts

• is there something like DownThemAll! that can be run without installing anything?
  1 project | reddit.com/r/Archiveteam | 16 Jan 2022
• What are all the websites that can access LBRY?
  2 projects | reddit.com/r/lbry | 14 Jan 2022
• How is ArchiveBox?
  4 projects | reddit.com/r/selfhosted | 27 Dec 2021
• Is there a web text extraction library for reader mode written in Java/Kotlin?
  1 project | reddit.com/r/webdev | 20 Dec 2021
• Best content extraction library from news link?
  3 projects | reddit.com/r/webscraping | 14 Dec 2021
• Python using Selenium to crawl data from Google Maps
  1 project | reddit.com/r/Python | 19 Nov 2021
• CuriosityStream Ripping
  1 project | reddit.com/r/DataHoarder | 27 Oct 2021

Index

Sponsored

Less time debugging, more time building

Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

scoutapm.com

Get the trending Python projects
with our weekly report!
» Subscribe «