library-of-alexandria VS Apache PDFBox

After a quick Google search, I figured out that only less than 1% of ancient texts survived to the modern day. This unfortunate fact was my inspiration to start working on an ambitious web crawling and archival project, called the Library of Alexandria.

Here is a link to the latest releases: https://github.com/bottomless-archive-project/library-of-alexandria/releases

What do you mean by storage system? Just curious because I'm working on a similar project.

If anybody is interested about the project mentioned in the interview, it's available here: https://github.com/bottomless-archive-project/library-of-alexandria

I'm collecting documents. Working on an app suite called Library of Alexandria. Got 91 million docs atm (mostly PDFs) and it's only going up. All of that fits on around 100 TB with gzip compression.

I'm working on an OSS called Library of Alexandria. It is an application that is built to collect, archive, and make searchable various (mostly PDF) documents. I have a little bit more than 90 million documents archived. My next step is to somehow label/classify them.

90 TB of PDFs. I'm working on the Library of Alexandria project. Just a fun little library, nothing more. 😅😅😅

Are you looking for a way to render PDF's or produce them? If you want to produce PDF's, I've used https://pdfbox.apache.org/ successfully as well as https://itextpdf.com/ (potentially costs money).

If you don't mind using java, you can use the open source Apache PDFBox library

https://pdfbox.apache.org/

It's relatively performant and it's a mature and supported codebase that can accomplish most pdf tasks.

Then, open the pdf_utils/pom.xml file and add a dependency to PDFBox, in the dependencies section:

There are some libraries for Java that can help you create a PDF file such as PDFBox or IText. Here there's a short exaplanation on how to use them.

So, when I started to view the documents, a lot of them simply failed to open. I had to look around for a library that could verify PDF documents. I had some experience with PDFBox in the past, so it seemed to be a good go-to solution. It had no way to verify documents by default, but it could open and parse them and that was enough to filter out the incorrect ones. It felt a little bit strange just to read the whole PDF into the memory to verify if it is correct or not, but hey I needed a simple fix for now and it worked really well.

PDFBox can do this. It’s not part of the CLI but it wouldn’t be too hard to add:

https://github.com/apache/pdfbox/blob/5b00807463279f1002e245...