library-of-alexandria VS precomp-cpp

After a quick Google search, I figured out that only less than 1% of ancient texts survived to the modern day. This unfortunate fact was my inspiration to start working on an ambitious web crawling and archival project, called the Library of Alexandria.

Here is a link to the latest releases: https://github.com/bottomless-archive-project/library-of-alexandria/releases

What do you mean by storage system? Just curious because I'm working on a similar project.

If anybody is interested about the project mentioned in the interview, it's available here: https://github.com/bottomless-archive-project/library-of-alexandria

I'm collecting documents. Working on an app suite called Library of Alexandria. Got 91 million docs atm (mostly PDFs) and it's only going up. All of that fits on around 100 TB with gzip compression.

I'm working on an OSS called Library of Alexandria. It is an application that is built to collect, archive, and make searchable various (mostly PDF) documents. I have a little bit more than 90 million documents archived. My next step is to somehow label/classify them.

90 TB of PDFs. I'm working on the Library of Alexandria project. Just a fun little library, nothing more. 😅😅😅

You should look at Precomp - an apparently abandoned, but awesomely working arhiver that is dedicated to compressing those ZIP-based formats, and compresses them twice-thrice as small as LZMA2 can.

You can also have a look at this packer. It compresses PDF and EPUB 2-3 times smaller than 7z at maximum settings, at half the speed. I keep all my books in it and never had a problem.

There is an opensource archiver called precomp that is designed specifically for e-documents and compresses them 2-3 times smaller than 7z in ultra mode. It works, and I keep my e-books in it. However, it seems nobody else does, and development has stopped due to lack of interest.

There are tools like preflate [1] or precomp [2] that guarantees a bitwise identical reconstruction, of course modulo bugs.

[1] https://github.com/deus-libri/preflate

[2] https://github.com/schnaader/precomp-cpp/ (which internally makes use of preflate)

You may use "precomp" in order to seek and uncompress those parts. Then you can use a better compression algorithm : http://schnaader.info/precomp.php

Precomp http://schnaader.info/precomp.php is a program which will seek compressed parts and decompress those parts in order to compress with another algorithm. This greatly improve compression ratio with the other algorithm.

This may be of help. (a255854 and d649919)