Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Heritrix3 Alternatives
Similar projects and alternatives to heritrix3 based on common topics and language
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better heritrix3 alternative or higher similarity.
heritrix3 reviews and mentions
Posts with mentions or reviews of heritrix3.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-07-06.
-
WARC'in the Crawler
If anyone's interested in web crawling technology, check out Heretrix [1], been around since 2004 and while not the most performant it has incorporated many responsible disciplines in the design and as this article pointed out, WARC format.
1. https://heritrix.readthedocs.io
-
Why isn't the internet more fun and weird?
I've considered running my own IA (https://github.com/internetarchive/heritrix3), which only archives my chrome history, which is stored in a local sqlite db.
I haven't had the time to figure out the details of how the pieces should be glued together.
-
Is there a way to archive groups of webpages similarly to how web archive does it?
You can go all the way and use the same tools as the Internet Archive: https://github.com/internetarchive/heritrix3
- Heritrix: Internet Archive's extensible, web-scale, archival-quality web crawler
- Best Http client for web scraping
-
The Internet Is Rotting
By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 https://iipc.github.io/warc-specifications/specifications/wa...).
The main crawler still seems to be heritrix3 (https://github.com/internetarchive/heritrix3), but there's a great little ecosystem with tools such as webrecorder and warcprox.
Still, I've read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens.
Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.
-
A note from our sponsor - InfluxDB
www.influxdata.com | 3 May 2024
Stats
Basic heritrix3 repo stats
6
2,691
6.2
22 days ago
internetarchive/heritrix3 is an open source project licensed under GNU General Public License v3.0 or later which is an OSI approved license.
The primary programming language of heritrix3 is Java.
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com