warcio VS pywb

What you're looking at are WARC (Web ARChive) files, which contain the raw API responses saved from YouTube. You need to parse them into usable data with something like warcio, then ingesting it into a database.

The tool you want is warcio which is available in pypi and has a command line interface as well. You can use that to extract contents, or scan through warcs, or build new warcs.

(Note: I use warcio for this, but the description above explains what is actually happening)

I have thousands of bookmarks on raindrop.io that I've been wanting to archive for a while. However, I've archived ~150 pages so far with Pywb and it ended up being 500MB across two WARCs, even with the dedupe setting specified in my settings file. It dedupes while archiving pages. I want software to get any spots missed and be sure that WARCs are actually deduped.

I don't know if the solution would be related or relevant to this, but I would also want to be able to remotely launch and access a web server, Pywb, on Safari on my iPad, also no matter what wifi I'm on. On a Mac, it would be launched with the command wayback and the server would be accessed on the Browser with localhost:8080.

Check their github site. I would try "git clone https://github.com/webrecorder/pywb `

I've been trying to open those large warc files these days. I've tried webrecorder, replayweb, pywb and warcat before but none of these worked well for me.

pywb, but it requires creating a full copy of the data: https://github.com/webrecorder/pywb/issues/408

I found it to be good with loading small WARCs quickly, but it can longer if the WARC is larger. Webarchive player, while it's old and discontinued, I've found it work better than Webrecorder Player and replayweb.page. If you want newer software to replay WARCs, try Pywb. I find it to be the best WARC player.

I use pywb in proxy recording mode.