warc2zim VS zim-requests

There are clearly similarities between the two, given that Kiwix put resources into making WARC content available in ZIM archives (i.e. Zimit-style ZIMs, created with the Zimit scraper and warc2zim backend). But as u/IMayBeABitShy said, the ZIM specification focuses on providing a highly compressed container that is readable on-the-fly (i.e. by decompressing only the needed content to show an article), whereas WARC, or rather the compressed version WACZ, is merely a zipped version of the WARC data (request headers and responses). It is also readable on-the-fly, but compression will not be as optimal as the zstandard compression used by modern ZIM archives.

Or, use the warcit tool on your collection of documents, and convert them to a warc archive, then use the warc2zim commandline tool to convert the WARC to a ZIM;

Well yes you can, but you will need to post the request at https://github.com/openzim/zim-requests/issues

I see their content is under a CC-license, so it's fair game for Kiwix. Open a ticket at https://github.com/openzim/zim-requests/issues to have it added to the queue.

As for scrapers of interest, well there's always some maintenance needed on the existing ones if you need a warmup of sorts, and I wouldn't mind it if mwoffliner was also able to pick up on wikia/fandom websites. Other than that on our current wishlist there's also code.org, or even manuallibs.com if you are really bad at design (as the ticket explains we might want the content but not the website design).

I see a request from last year, and seems there are/were throttling issues with it. There is currently a 1GB .zim file in the linked folder though.

To submit this to the kiwix library, your best option would probably to create an issue in the zim-requests github repo.

Open a ticket at https://github.com/openzim/zim-requests/issues - if there's an off-the-shelf solution for the website you requested then it'll be added to the zimfarm/library.

Yo, you know you can request specific websites to be scraped at https://github.com/openzim/zim-requests/issues ?

I don't think that's possible as of yet but 1. There is a ticket for ReadtheDocs https://github.com/openzim/zim-requests/issues/446

Not that I know, but you can ask for confirmation on r/kiwix or if you have a specific website in mind then file a request on the zim-requests github repository.

The help we need there is to have someone scout around the internets, list resources that are up for the taking (usually Creative Commons-licensed, in one flavour or another) and open a zim-request on github.