webmagic vs fscrawler

Our great sponsors

InfluxDB - Power Real-Time Data Analytics at Scale

WorkOS - The modern identity platform for B2B SaaS

SaaSHub - Software Alternatives and Reviews

Our great sponsors

webmagic		fscrawler
	Project
1	Mentions	3
11,239	Stars	1,295
-	Growth	-
7.0	Activity	9.4
4 days ago	Latest Commit	12 days ago
Java	Language	Java
Apache License 2.0	License	Apache License 2.0

The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

webmagic

Posts with mentions or reviews of webmagic. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-01-24.

What's the most interesting open-source project to study?
13 projects | /r/learnjava | 24 Jan 2022

webmagic

fscrawler

Posts with mentions or reviews of fscrawler. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-10-08.

Tools for converting assorted text files to JSON?
2 projects | /r/elasticsearch | 8 Oct 2022

Or use the ingest Attachment plugin, or https://github.com/dadoonet/fscrawler and remove any doubt
How to index PDF files with AppSeach
1 project | /r/elasticsearch | 9 Jan 2021

Did you sir give FSCrawler a shot? it's also a file system crawler that automatically ingests files into an ElasticSearch index and it is built over Tika so it performs typically the same (multiple file formats, multiple languages, ...). It can also perform OCR while indexing (uses Tesseract) by just toggling it on its config (I didn't try the OCR that much for a lack of need and for the ridiculous amount of times it added to the indexation process on my small environment).

What are some alternatives?

When comparing webmagic and fscrawler you can also consider the following projects:

Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.

Elasticsearch - Free and Open, Distributed, RESTful Search Engine

google-search-results-java - Google Search Results JAVA API via SerpApi

mutad - Realtime Data Processing and Search Engine Implementation.

Flowable (V6) - A compact and highly efficient workflow and Business Process Management (BPM) platform for developers, system admins and business users.

nifi-extracttext-processor - Apache NiFi Custom Processor Extracting Text From Files with Apache Tika

spring-cloud-config - External configuration (server and client) for Spring Cloud

analysis-ik - 🚌 The IK Analysis plugin integrates Lucene IK analyzer into Elasticsearch and OpenSearch, support customized dictionary.

ActiveJ - ActiveJ is an alternative Java platform built from the ground up. ActiveJ redefines core, web and high-load programming in Java, providing simplicity, maximum performance and scalability

advanced-java - 😮 Core Interview Questions & Answers For Experienced Java(Backend) Developers | 互联网 Java 工程师进阶知识完全扫盲：涵盖高并发、分布式、高可用、微服务、海量数据处理等领域知识

Arthas - Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas

url-collector - An application that crawls the Common Crawl corpus for URLs with the specified file extensions.