The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 web-scraper Open-Source Projects
-
Project mention: More than 400 start.me OSINT websites! More than 10KB of sources! | /r/OSINT | 2023-04-11
-
Work on a personal project. There's a list of 100 sample projects at https://github.com/arpit-omprakash/100ProjectsOfCode
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
Use Lightnovel crawler on a computer in terminal or in their discord bot to find series across multiple LN / webnovel sites then choose the format to download (epub,pdf, txt, and many more)
-
Project mention: Ask HN: Most interesting tech you built for just yourself? | news.ycombinator.com | 2023-04-27
Two years ago I decided to built my own web browser, with the underlying idea to use the internet more efficiently (and to force cache everything).
Took a while to find the architecture, but it's still an unfinished ambitious project. You can probably spend forever working on HTML and CSS fixes alone...
-
Monkey-DL (Anime Downloader)
Bulk download your favourite anime episodes from your favourite anime websites
-
spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use. (by postmodern)
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
web-scraping
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
-
-
google-maps-scraper
scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place (by gosom)
-
-
summarizer
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
-
facebook_page_scraper
Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV
-
-
It's been a cool learning experience making a Product Hunt listing, a small demo video, and allll the social posts (Twitter, LinkedIn, etc).
-
So it's this https://github.com/gan-of-culture/get-sauce ?
-
Project mention: AI Report #4: AutoGPT And Open-source lags behind Part 2 | news.ycombinator.com | 2023-06-15
> The google search function is also limited. For comparison, SerpAPI masterfully scrapes Google Search using a proxy network and very intelligent parsing. In experiments using SerpAPI in combination with Microsoft’s guidance module, I got much farther than AutoGPT.
Thanks for your kind words. We are working on SerpApi integration for Auto-GPT: https://github.com/serpapi/public-roadmap/issues/905
-
CobWeb-lnx
CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.
-
Project mention: [OpenSource] I am building high performance Plex alternative in Go for Movies and TV Show | /r/golang | 2023-06-02
I also build a similar tool, it let's you choose and play movies. I used webtorrent behind the scenes. https://github.com/qascade/yast
-
-
tagalog-dictionary-scraper
Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com
-
mexican-jobs-2020
Data ETL & Analysis on thousands of job listings from the official Mexican job board (2020 edition).
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
web-scraper related posts
- Show HN: A Google Maps Scraper
- Google Maps Scraper in Golang
- I'm trying and failing to compile someone else's project to wasm.
- Help with Paperback IOS.
- Fired from an internship after 2 weeks
- Need help thinking of a personal project
- Multiparadigmatic Web Scraping Tool!
-
A note from our sponsor - WorkOS
workos.com | 28 Mar 2024
Index
What are some of the best open-source web-scraper projects? This list will help you:
Project | Stars | |
---|---|---|
1 | awesome-crawler | 6,023 |
2 | 100ProjectsOfCode | 2,832 |
3 | soup | 2,125 |
4 | lightnovel-crawler | 1,258 |
5 | stealth | 986 |
6 | Monkey-DL (Anime Downloader) | 804 |
7 | spidr | 788 |
8 | web-scraping | 617 |
9 | PHP Scraper | 487 |
10 | google-maps-scraper | 469 |
11 | basketball_reference_web_scraper | 397 |
12 | summarizer | 267 |
13 | awesome-web-scraper | 231 |
14 | facebook_page_scraper | 183 |
15 | cascadia | 134 |
16 | Senpwai | 116 |
17 | get-sauce | 109 |
18 | public-roadmap | 43 |
19 | CobWeb-lnx | 38 |
20 | yast | 28 |
21 | reddit-bots | 23 |
22 | tagalog-dictionary-scraper | 22 |
23 | mexican-jobs-2020 | 21 |