java-warc VS jsoup

I found one Java library on Github (thanks Mixnode) that was able to read these files. Unfortunately, it was not maintained for the past couple of years. I picked it up and forked it to make it a little easier to use. (A couple of years later this repo was moved under the Bottomless Archive project as well.)

2e. The HTML parsing library is in app-kt. It's called JSoup https://jsoup.org/

For parsing the webpage in Java or Kotlin you can use Jsoup

Jsoup — It is a Java library that can be used for both extracting and parsing HTML.

Finally, at this point, I was able to go through a bunch of webpages (parsing them in the process with JSoup), grab all the links that contained pdf files based on the file extension then download them. Unsurprisingly, most of the pages (~60-80%) ended up being unavailable (404 Not Found and friends). After a quick cup of coffee, I got the 10.000 documents on my hard drive. This is when I realized that I have one more problem to solve.

It might be better to use an HTML parser instead (this one looks good at first glance: https://jsoup.org/) although as long as you can make certain assumptions over the HTML input (for example that it'll always have those two attributes in this order) using regular expressions to parse it is feasible.

Poti folosi JSoup in Java https://jsoup.org/

There are five machine problems. None of the assignments took me longer than two to three hours, and the last one I completed in less than an hour. The MPs had been recently redesigned and tied together nicely. Each one covered a different course topic in the jsoup code base.

https://github.com/jhy/jsoup jsoup is a java library for parsing HTML. Intuitive API and extremely well readable code. I would definitely recommend this.