pywb
warcio
pywb | warcio | |
---|---|---|
7 | 4 | |
1,303 | 345 | |
1.2% | 1.7% | |
7.2 | 2.5 | |
15 days ago | 12 days ago | |
JavaScript | Python | |
GNU General Public License v3.0 only | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pywb
-
Is there any good software for deduping (deduplicating) content in WARC files?
I have thousands of bookmarks on raindrop.io that I've been wanting to archive for a while. However, I've archived ~150 pages so far with Pywb and it ended up being 500MB across two WARCs, even with the dedupe setting specified in my settings file. It dedupes while archiving pages. I want software to get any spots missed and be sure that WARCs are actually deduped.
-
Is there a way to easily and reliably SSH to my laptop no matter what wifi the laptop is connected to? I have no clue.
I don't know if the solution would be related or relevant to this, but I would also want to be able to remotely launch and access a web server, Pywb, on Safari on my iPad, also no matter what wifi I'm on. On a Mac, it would be launched with the command wayback and the server would be accessed on the Browser with localhost:8080.
-
I can't install a Python package, pywb, looks like a problem with brotlipy. What can I do?
Check their github site. I would try "git clone https://github.com/webrecorder/pywb `
-
Purevolume archives?
I've been trying to open those large warc files these days. I've tried webrecorder, replayweb, pywb and warcat before but none of these worked well for me.
-
Ran grab-site now have some warc.gz files etc, the site in question was originally hosted in a mixture of html and javascript, what's the best and easiest way to make this accessible as a user for offline personal use?
pywb, but it requires creating a full copy of the data: https://github.com/webrecorder/pywb/issues/408
-
How good is ArchiveWeb.page?
I found it to be good with loading small WARCs quickly, but it can longer if the WARC is larger. Webarchive player, while it's old and discontinued, I've found it work better than Webrecorder Player and replayweb.page. If you want newer software to replay WARCs, try Pywb. I find it to be the best WARC player.
-
Saving all browsed websites automatically
I use pywb in proxy recording mode.
warcio
-
Can anyone familiar with databases of Youtube archives help me? I don't know how to find what I'm looking for. Details in post.
What you're looking at are WARC (Web ARChive) files, which contain the raw API responses saved from YouTube. You need to parse them into usable data with something like warcio, then ingesting it into a database.
- Any very noob friendly way to extract images and videos from WARC files?
-
Help with WARC files/Extracting a portion of a full crawl
The tool you want is warcio which is available in pypi and has a command line interface as well. You can use that to extract contents, or scan through warcs, or build new warcs.
-
Is anyone working on a Yahoo Answers Archive yet, and if so where can we go to find it?
(Note: I use warcio for this, but the description above explains what is actually happening)
What are some alternatives?
conifer - Collect and revisit web pages.
ArchiveBox - 🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
awesome-selfhosted - A list of Free Software network services and web applications which can be hosted on your own servers
youtube-discussions-grab
replayweb.page - Serverless replay of web archives directly in the browser
yahoo-answers-archiveteam-compose
SingleFileZ - Web Extension to save a faithful copy of an entire web page in a self-extracting ZIP file
youtube-discussions-archive - EXPERIMENTAL YouTube Discussion Tab Downloader
22120 - 💾 Diskernet - Your preferred backup solution. It's like you're still online! Full text search archive from your browsing and bookmarks. Weclome! to the Diskernet: an internet on yer disk. Disconnect with Diskernet, an internet for the post-online apocalypse. Or the airplane WiFi. Or the site goes down. Or ... You get the picture. Get Diskernet. 80s logo. Formerly 22120 (project codename) ;P ;) xx;p [Moved to: https://github.com/i5ik/Diskernet]
webarchiveplayer - NOTE: This project is no longer being actively developed.. Check out Webrecorder Player for the latest player. https://github.com/webrecorder/webrecorderplayer-electron) (Legacy: Desktop application for browsing web archives (WARC and ARC)
webrecorder-player - Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)
sandcrawler - Backend, IA-specific tools for crawling and processing the scholarly web. Content ends up in https://fatcat.wiki