fscrawler vs url-collector

Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

www.influxdata.com

featured

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

featured

fscrawler		url-collector
	Project
3	Mentions	2
1,308	Stars	0
-	Growth	-
9.4	Activity	5.1
2 days ago	Latest Commit	over 2 years ago
Java	Language	Java
Apache License 2.0	License	MIT License

The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

fscrawler

Posts with mentions or reviews of fscrawler. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2022-10-08.

Tools for converting assorted text files to JSON?
2 projects | /r/elasticsearch | 8 Oct 2022

Or use the ingest Attachment plugin, or https://github.com/dadoonet/fscrawler and remove any doubt
How to index PDF files with AppSeach
1 project | /r/elasticsearch | 9 Jan 2021

Did you sir give FSCrawler a shot? it's also a file system crawler that automatically ingests files into an ElasticSearch index and it is built over Tika so it performs typically the same (multiple file formats, multiple languages, ...). It can also perform OCR while indexing (uses Tesseract) by just toggling it on its config (I didn't try the OCR that much for a lack of need and for the ridiculous amount of times it added to the indexation process on my small environment).

url-collector

Posts with mentions or reviews of url-collector. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2021-10-02.

240 million URLs for PDF and DOC files
3 projects | /r/DataHoarder | 2 Oct 2021

Well, I used Java. The app is still somewhat under construction, but it is available here: https://github.com/bottomless-archive-project/url-collector

What are some alternatives?

When comparing fscrawler and url-collector you can also consider the following projects:

Elasticsearch - Free and Open, Distributed, RESTful Search Engine

library-of-alexandria - Library of Alexandria (LoA in short) is a project that aims to collect and archive documents from the internet.

mutad - Realtime Data Processing and Search Engine Implementation.

SpotifyDiscoveryBot - A Java-based bot that automatically crawls for new releases by your followed artists on Spotify. Never miss a release again!

webmagic - A scalable web crawler framework for Java.

nifi-extracttext-processor - Apache NiFi Custom Processor Extracting Text From Files with Apache Tika

analysis-ik - 🚌 The IK Analysis plugin integrates Lucene IK analyzer into Elasticsearch and OpenSearch, support customized dictionary.

advanced-java - 😮 Core Interview Questions & Answers For Experienced Java(Backend) Developers | 互联网 Java 工程师进阶知识完全扫盲：涵盖高并发、分布式、高可用、微服务、海量数据处理等领域知识

appbase-droid - Elasticsearch and appbase.io library for Android (and Java)