Python web-scraper

Open-source Python projects categorized as web-scraper
Edit details
Topics: #Python #web-scraping #Scraper #Python3 #reddit-bot
Access the most powerful time series database as a service
Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
www.influxdata.com
Sponsored

Top 17 Python web-scraper Projects

  • lightnovel-crawler

    Generate and download e-books from online sources.

    Project mention: ISSTH left me disappointed | /r/noveltranslations | 2023-05-01

    For epubs, same for me. I use https://github.com/dipu-bd/lightnovel-crawler which allows you to make epub for many websites.

  • Monkey-DL (Anime Downloader)

    Bulk download your favourite anime episodes from your favourite anime websites

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • onlyfans-dl

    OnlyFans content downloader

  • web-scraping

    Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

    Project mention: web-scraping: NEW Data - star count:481.0 | /r/algoprojects | 2023-05-27

  • summarizer

    A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.

  • facebook_page_scraper

    Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV

  • CobWeb-lnx

    CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.

    Project mention: Multiparadigmatic Web Scraping Tool! | /r/computerscience | 2023-05-14

    PyPi: pypi.org/project/CobWeb-lnx/

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • tagalog-dictionary-scraper

    Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com

  • mexican-jobs-2020

    Data ETL & Analysis on thousands of job listings from the official Mexican job board (2020 edition).

  • reddit-bots

    A collection of Reddit bots that I use to enhance the subreddits I manage.

    Project mention: Any ideas? I need a bot to grab comments from a reddit post and put them on github repository | /r/github | 2023-01-05

    It's not what you're directly looking for but as an example and starting point I'd check out https://github.com/PhantomInsights/reddit-bots

  • tweet-transcriber

    A Reddit bot that transcribes tweets from comments and submissions links, mirrors their images and replies back with a formatted Markdown message.

  • git-pull

    Parallelized web scraper for Github

  • Python-Web-Scraper

    An adaptive Python Web Scraper App to catch the best deals by scraping and parsing data from select E-Commerce sites.

    Project mention: Python Web Scraper/Crawler for E-Commerce sites. Currently supports only a few websites but im looking to expand that list. Tips/criticism are welcomed. This is the first project for my student CV (0 working experience) so I'd like it to be as polished as possible. | /r/programming | 2023-03-01

  • Abosar

    অবসর 📚 A Collection Of Short Bengali Stories Web Scraped From Various Bengali eMagazines And eNewspapers.

  • varieteebot

    A telegram bot that sends today's tee of some tee shops.

  • nanoscrape

    Simple scraping program that can download webpages, Discord embeds, and more.

  • iw-scraper

    Web scraper for imovelweb listings

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-05-27.

Python web-scraper related posts

Index

What are some of the best open-source web-scraper projects in Python? This list will help you:

Project Stars
1 lightnovel-crawler 959
2 Monkey-DL (Anime Downloader) 744
3 onlyfans-dl 646
4 web-scraping 468
5 summarizer 259
6 facebook_page_scraper 117
7 CobWeb-lnx 31
8 tagalog-dictionary-scraper 21
9 mexican-jobs-2020 21
10 reddit-bots 21
11 tweet-transcriber 19
12 git-pull 15
13 Python-Web-Scraper 10
14 Abosar 5
15 varieteebot 3
16 nanoscrape 0
17 iw-scraper 0

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com