Top 23 scraping-website Open-Source Projects

• Ferret

  0 5,616 3.1 Go

  Declarative web scraping

  

• cloudflare-scrape

  3 3,291 0.0 Python

  A Python module to bypass Cloudflare's anti-bot page.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Crawly

  2 839 5.2 Elixir

  Crawly, a high-level web crawling & scraping framework for Elixir.

  

Project mention: Crawly – Elixir web scraping framework | news.ycombinator.com | 2023-08-26

• Edu-Mail-Generator

  1 788 0.0 Python

  Generate Free Edu Mail(s) within minutes

• google-maps-scraper

  3 703 5.6 Python

  👋 HOLA 👋 HOLA 👋 HOLA ! ENJOY OUR GOOGLE MAPS SCRAPER 🚀 TO EFFORTLESSLY EXTRACT DATA SUCH AS NAMES, ADDRESSES, PHONE NUMBERS, REVIEWS, WEBSITES, AND RATINGS FROM GOOGLE MAPS WITH EASE! 🤖

  

Project mention: I create a google maps scraper, let me know your thoughts | /r/webscraping | 2023-07-06

My scrapers runs at 120 Listing per 10 Minutes. So yours is quite Fast. You can see my scraper at https://github.com/omkarcloud/google-maps-scraper. It is quite popular with 95 Stars.

• Dataflow kit

  0 636 0.0 Go

  Extract structured data from web sites. Web sites scraping.

• sarenka

  1 614 0.0 Python

  OSINT tool - gets data from services like shodan, censys etc. in one app

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• PHP Scraper

  1 493 5.3 PHP

  A universal web-util for PHP.

• crawler

  0 298 8.1 PHP

  Library for Rapid (Web) Crawler and Scraper Development (by crwlrsoft)

  

• requests-html

  2 266 0.0

  Pythonic HTML Parsing for Humans™ (by kennethreitz)

• ebayScraper

  7 178 0.0 Python

  Scrape all eBay sold listings to determine average/median pricing, plot listings over time with trend lines, and extract to excel

• SpyScrap

  2 142 0.0 Python

  CLI and GUI for OSINT. Are you very exhibited on the Internet? Check it! Twitter, Tinder, Facebook, Google, Yandex, BOE. It uses facial recognition to provide more accurate results.

• scraply

  1 126 0.0 Go

  Scraply a simple dom scraper to fetch information from any html based website

• html2rss

  0 111 4.9 Ruby

  📰 Build RSS 2.0 feeds from websites (and JSON APIs) with a few CSS selectors.

  

• zippyshare-scraper

  1 108 0.7 Python

  A module to get direct downloadable links from zippyshare download page.

• spidergram

  3 99 8.5 TypeScript

  Structural analysis tools for complex web sites

  

Project mention: Data Scrape Tool for Existing Designs? | /r/UXDesign | 2023-05-04

• amazon_scraper

  1 75 0.0 JavaScript

  Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

  

• TeslaServiceManualScraper

  4 61 0.0 Python

  This script will download the Tesla Service Manual onto a local doc folder for offline access.

• outscraper-python

  3 61 6.9 Python

  The library provides convenient access to the Outscraper API from applications written in the Python language. Allows using Outscraper's services from your code.

  

Project mention: outscraper-python VS google-maps-scraper - a user suggested alternative | libhunt.com/r/outscraper-python | 2023-05-14

• torchestrator

  2 43 0.0 Kotlin

  Spin up Tor containers and then proxy HTTP requests via these Tor instances

• proxy_web_crawler

  3 41 7.5 Python

  Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords

• clicknium-samples

  6 24 0.0 Python

  Clicknium Python automation Samples

  

• FirstCyclingAPI

  1 22 4.0 Python

  An unofficial Python API wrapper for firstcycling.com

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

scraping-websites related posts

• Data Scrape Tool for Existing Designs?
  1 project | /r/UXDesign | 4 May 2023
• Sitemap Generator/Crawl for an Authenticated website
  1 project | /r/UXDesign | 23 Mar 2023
• Buscador de vídeos con OpenSearch y React | Parte 3 | Limpieza y almacenamiento de los datos
  1 project | dev.to | 18 Nov 2022
• Bypassing Cloudflare and ModSec checks
  1 project | /r/changedetectionio | 21 Oct 2022
• Build a instagram-scraper-it on github | free license
  1 project | dev.to | 3 Sep 2022
• Tesla Service Manuals (Google Drive)
  1 project | /r/opendirectories | 25 May 2022
• Tesla Service Manuals are now FREE. Get yours while you can!
  1 project | /r/teslamotors | 25 May 2022
• A note from our sponsor - WorkOS
  workos.com | 24 Apr 2024

  The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com