Java warc Projects

heritrix3

6 2,700 6.2 Java

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Project mention: WARC'in the Crawler | news.ycombinator.com | 2023-12-21

If anyone's interested in web crawling technology, check out Heretrix [1], been around since 2004 and while not the most performant it has incorporated many responsible disciplines in the design and as this article pointed out, WARC format.
1. https://heritrix.readthedocs.io

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Java warc related posts

WARC'in the Crawler

1 project | news.ycombinator.com | 21 Dec 2023
Is there a way to archive groups of webpages similarly to how web archive does it?

2 projects | /r/DataHoarder | 29 Mar 2022
Heritrix: Internet Archive's extensible, web-scale, archival-quality web crawler

1 project | news.ycombinator.com | 26 Sep 2021
Best Http client for web scraping

1 project | /r/java | 26 Sep 2021