JavaScript Crawler

Open-source JavaScript projects categorized as Crawler

Top 22 JavaScript Crawler Projects

  1. EasySpider

    A visual no-code/code-free web crawler/spider易采集:一个可视化浏览器自动化测试/数据采集/爬虫软件,可以无代码图形化的设计和执行爬虫任务。别名:ServiceWrapper面向Web应用的智能化服务封装系统。

    Project mention: EasySpider: A No-Code Tool for Visual Web Crawling and Data Collection | news.ycombinator.com | 2024-08-11
  2. SurveyJS

    JavaScript Form Builder with No-Code UI & Built-In JSON Schema Editor. Add the SurveyJS white-label form builder to your JavaScript app (React/Angular/Vue3). Build complex JSON forms without coding. Fully customizable, works with any backend, perfect for data-heavy apps. Learn more.

    SurveyJS logo
  3. browser-fingerprinting

    Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

  4. work_crawler

    Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.

  5. google-play-scraper

    Node.js scraper to get data from Google Play

  6. article-extractor

    To extract main article from given URL with Node.js

    Project mention: Show HN: I built an AI satirical news site because news was depressing me | news.ycombinator.com | 2025-02-06

    Actually, I kept it simple - I use the original images from the news articles! When I fetch an article through RSS and extract its content using the @extractus/article-extractor library, it pulls the main image along with the content.

    https://github.com/extractus/article-extractor

  7. single-file-cli

    CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

    Project mention: Omnom: Self-hosted bookmarking with searchable, wysiwyg snapshots [showcase] | news.ycombinator.com | 2025-04-14

    Alternatively, you can also use the SingleFile extension to snapshot what you want and upload it in place of the automated snapshot. This is also handy because the extension allows you to remove private data prior to screenshot, such as your name or username.

    I personally have cookies in place for most common social media sites that need login (twitter, reddit), and if I need to snapshot something else occasionally, I do it manually and upload it to Linkding.

    https://linkding.link/

    https://linkding.link/archiving/

    https://www.getsinglefile.com/

  8. rebrowser-patches

    Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

    Project mention: Rebrowser Patches – Patches for undetectable browser automation | news.ycombinator.com | 2025-04-25
  9. Civic Auth

    Auth in Less Than 5 Minutes. Civic Auth comes with multiple SSO options, optional embedded wallets, and user management — all implemented with just a few lines of code. Start building today.

    Civic Auth logo
  10. sitemap-generator

    Easily create XML sitemaps for your website.

  11. JSSoup

    JavaScript + BeautifulSoup = JSSoup

  12. th-music-video-generator

    Touhou Project random music video generator/player, crawling image and video from websites to generate MV.

  13. spiderable-middleware

    Pre-rendering for JavaScript websites that delivers SSR-level SEO, enhanced link previews, and performance via effortless middleware integration — ideal for PWAs, SPAs, and modern JS-driven apps, websites, and webpages

    Project mention: 📦 spiderable-middleware update | dev.to | 2025-02-12

    Read full changelog here

  14. undetectable-crawler

    A Node.js script powered by Puppeteer for undetectable web scraping

  15. selector-finder

    Find a CSS selector on a public site

  16. images-downloader

    A Node.js module for downloading a single image or multiple images to disk from a given Url

  17. Studybyte

    Studybyte is a search engine designed to help students find educational content effortlessly.

  18. socialblade-com-api

    Unofficial APIs for socialblade.com website.

  19. CodexDrake

    An open source, privacy-first, self-hosting capable and blazing fast search engine written in JavaScript. Browse anonymously and safely without the need to pay third-party APIs. 👀

  20. airbnb-scraper

    Apify public actor for scraping Airbnb homes.

  21. finance-news-crawler

    Finance News Crawler uses News API to fetch some latest articles and generates a sentiment report with the OpenAI API or VADER

  22. Netflix-Hotkeys

    A Chrome extension to enhance your Netflix binging experience!

  23. tumblweed

    A simple cross-platform Tumblr blog downloader

  24. dora-cli

    A CLI version for the deep search tool.

    Project mention: CLI Semantic Search | news.ycombinator.com | 2025-03-26
  25. InfluxDB

    InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

    InfluxDB logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

JavaScript Crawler discussion

Log in or Post with

JavaScript Crawler related posts

  • Cool ChatGPT Finance Sentiment Analysis

    1 project | /r/ChatGPT | 2 Jul 2023
  • GitHub - simwai/finance-news-crawler: Finance News Crawler uses News API to fetch some latest articles and generates a sentiment report with the OpenAI API or VADER

    1 project | /r/algotrading | 2 Jul 2023
  • Would it be worth publishing my Chrome Extension even if I anticipate that it will have very few users?

    1 project | /r/developersIndia | 16 Jun 2023
  • Who here is developing extensions?

    2 projects | /r/webdev | 11 Jun 2023
  • Netflix Hotkeys: A Chrome Extension to enhance your Netflix Experience

    1 project | /r/javascript | 11 Jun 2023
  • Netflix Hotkeys: A Chrome Extension to enhance your Netflix Experience

    1 project | /r/coolgithubprojects | 10 Jun 2023
  • FAQs on my side project

    2 projects | /r/SideProject | 24 Oct 2022
  • A note from our sponsor - SurveyJS
    surveyjs.io | 12 May 2025
    Add the SurveyJS white-label form builder to your JavaScript app (React/Angular/Vue3). Build complex JSON forms without coding. Fully customizable, works with any backend, perfect for data-heavy apps. Learn more. Learn more →

Index

What are some of the best open-source Crawler projects in JavaScript? This list will help you:

# Project Stars
1 EasySpider 38,774
2 browser-fingerprinting 4,287
3 work_crawler 3,417
4 google-play-scraper 2,474
5 article-extractor 1,700
6 single-file-cli 800
7 rebrowser-patches 741
8 sitemap-generator 431
9 JSSoup 369
10 th-music-video-generator 274
11 spiderable-middleware 40
12 undetectable-crawler 30
13 selector-finder 26
14 images-downloader 20
15 Studybyte 16
16 socialblade-com-api 16
17 CodexDrake 12
18 airbnb-scraper 11
19 finance-news-crawler 11
20 Netflix-Hotkeys 9
21 tumblweed 6
22 dora-cli 5

Sponsored
JavaScript Form Builder with No-Code UI & Built-In JSON Schema Editor
Add the SurveyJS white-label form builder to your JavaScript app (React/Angular/Vue3). Build complex JSON forms without coding. Fully customizable, works with any backend, perfect for data-heavy apps. Learn more.
surveyjs.io

Did you know that JavaScript is
the 3rd most popular programming language
based on number of references?