Top 11 TypeScript Crawler Projects

crawlee

30 12,222 9.8 TypeScript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Project mention: Crawlee · Build reliable crawlers. Fast | news.ycombinator.com | 2024-05-08

firecrawl

3 2,636 7.5 TypeScript

🔥 Turn entire websites into LLM-ready markdown

Project mention: Show HN: Extracting structured data from the web with LLMs | news.ycombinator.com | 2024-05-01

SurveyJS

surveyjs.io featured

Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
x-crawl

8 1,230 9.3 TypeScript

x-crawl is a flexible Node.js multifunctional crawler library. Flexible usage and numerous functions can help you quickly, safely, and stably crawl pages, interfaces, and files.

Project mention: Flexible Node.js AI-assisted crawler library | news.ycombinator.com | 2024-04-24

algoliasearch-netlify

1 260 1.8 TypeScript

Official Algolia Plugin for Netlify. Index your website to Algolia when deploying your project to Netlify with the Algolia Crawler

Project mention: Adding Algolia search to my 404 page | dev.to | 2023-08-31

I then tried to install the @algolia/algoliasearch-netlify-frontend plugin, but the install is broken on Windows because they're using a UNIX specific command in their postinstall script. I started off by including it from JSDelivr instead as per their docs, but ran into some issues with not being able to use the head property on the Nuxt error layout.

npm-search

1 128 6.6 TypeScript

🗿 npm ↔️ Algolia replication tool :skier: :snail: :artificial_satellite: (by algolia)
extension

1 58 0.0 TypeScript

web scraping extension (by get-set-fetch)
billboard-json

4 30 9.4 TypeScript

🎧 Get json type billboard hot 100 chart
InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
retro-env-can-weather-chan

5 27 9.3 TypeScript

Retro Environment Canada Weather Channel for your browser
metafy-svg

1 13 0.0 TypeScript

Easily crawl a website's metadata and generate SVG as a service.
MicroFrontier

1 7 0.0 TypeScript

A lightweight crawler frontier implementation in TypeScript using Redis.
recrawl

1 1 0.0 TypeScript

Filesystem crawler

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

TypeScript Crawler related posts

Show HN: Extracting structured data from the web with LLMs

2 projects | news.ycombinator.com | 1 May 2024
Flexible Node.js AI-assisted crawler library

3 projects | news.ycombinator.com | 24 Apr 2024
Traditional crawler or AI-assisted crawler? How to choose?

1 project | dev.to | 22 Apr 2024
Tutorial: Extracting structured data from websites using Groq and Firecrawl

1 project | news.ycombinator.com | 22 Apr 2024
AI+Node.js x-crawl crawler: Why are traditional crawlers no longer the first choice for data crawling?

1 project | dev.to | 16 Apr 2024
AI combined with Node.js x-crawl crawler

1 project | dev.to | 10 Apr 2024
Recommend a flexible Node.js multi-functional crawler library —— x-crawl

1 project | dev.to | 20 Mar 2024
A note from our sponsor - SurveyJS
surveyjs.io | 8 May 2024

With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js. Learn more →

Index

What are some of the best open-source Crawler projects in TypeScript? This list will help you:

	Project	Stars
1	crawlee	12,222
2	firecrawl	2,636
3	x-crawl	1,230
4	algoliasearch-netlify	260
5	npm-search	128
6	extension	58
7	billboard-json	30
8	retro-env-can-weather-chan	27
9	metafy-svg	13
10	MicroFrontier	7
11	recrawl	1