Crawler4j vs storm-crawler

Crawler4j

Open Source Web Crawler for Java (by yasserg)

Web Crawling

Source Code

Suggest alternative

Edit details

storm-crawler

A scalable, mature and versatile web crawler based on Apache Storm (by apache)

Web Crawling web-crawler apache-storm Distributed Java Crawler stormcrawler

Source Code

stormcrawler.apache.org

Suggest alternative

Edit details

InfluxDB - Power Real-Time Data Analytics at Scale

Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

www.influxdata.com

featured

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

featured

Crawler4j		storm-crawler
	Project
2	Mentions	-
4,493	Stars	855
-	Growth	0.8%
0.0	Activity	8.8
over 2 years ago	Latest Commit	8 days ago
Java	Language	HTML
Apache License 2.0	License	Apache License 2.0

The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

Crawler4j

Posts with mentions or reviews of Crawler4j. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2021-03-02.

Technical View: Crawler and NLP to do website text analytics
1 project | dev.to | 11 Aug 2022

To do web crawling you can choose a tool in Java or Python. In my case I'm using Crawler4J. (https://github.com/yasserg/crawler4j).
Any decent web crawler?
4 projects | /r/rust | 2 Mar 2021

and Java github.com/yasserg/crawler4j

storm-crawler

Posts with mentions or reviews of storm-crawler. We have used some of these posts to build our list of alternatives and similar projects.

We haven't tracked posts mentioning storm-crawler yet.
Tracking mentions began in Dec 2020.

What are some alternatives?

When comparing Crawler4j and storm-crawler you can also consider the following projects:

jsoup - jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.

Apache Nutch - Apache Nutch is an extensible and scalable web crawler

Sparkler - Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

colly - Elegant Scraper and Crawler Framework for Golang

PeARS-orchard - This is the development version of PeARS, the people's search engine. More compact but less robust than PeARS-lite. If you just want to use PeARS as a local indexer, use PeARS-lite instead.

voyager - crawl and scrape web pages in rust

scaling-to-distributed-crawling - Repository for the Mastering Web Scraping in Python: Scaling to Distributed Crawling blogpost with the final code.

spider - The fastest web crawler written in Rust. Maintained by @a11ywatch.

crawlee - Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.