tubeup
heritrix3
tubeup | heritrix3 | |
---|---|---|
15 | 6 | |
383 | 2,691 | |
2.6% | 1.0% | |
6.4 | 6.2 | |
14 days ago | 22 days ago | |
Python | Java | |
GNU General Public License v3.0 only | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tubeup
- How can I archvie a youtube video?
-
Advanced and easy YouTube archiver now stable
If you want to provide an option to upload artifacts to the Internet Archive, you could crib off of https://github.com/bibanon/tubeup
- Swedish amateur archivist “Rosa Mannen” shutdown from YouTube, again
- 52% of YouTube videos live in 2010 have been deleted
-
Notice regarding Termination of Our Contract with “Uruha Rushia”
ArchiveTeam recommends the use of TubeUp.py, which acts as a wrapper for yt-dl to use for comprehensive archival, in addition to automatically uploading the content to the Internet Archive.
-
Downloading YouTube Private Videos
What I do is use a tool called tubeup. It will download a YouTube link you give it and automatically upload it to archive.org. It can take single videos or even whole playlists. That way, if it gets removed from YouTube, I know where to find it again, and I don't have to manage local copies or use up own cloud storage.
-
Am I actually wasting time & effort by archiving a still existing Youtube channel?
This can automate a LOT of the manual work: https://github.com/bibanon/tubeup
-
Is There A Youtube Liked Videos Saver?
You could pass a playlist of your liked videos to https://github.com/bibanon/tubeup
- Backing up a YouTube channel with rare live recordings
-
How to archive YouTube videos on the Wayback machine
I have tried the tubeup project but I spent most of the day trying to make that work but to no avail (I am working on macos Big Sur). I have also seen this post from what seems to be one of the staff, but it seems like a very long and complicated process to take just to archive YouTube videos.
heritrix3
-
WARC'in the Crawler
If anyone's interested in web crawling technology, check out Heretrix [1], been around since 2004 and while not the most performant it has incorporated many responsible disciplines in the design and as this article pointed out, WARC format.
1. https://heritrix.readthedocs.io
-
Why isn't the internet more fun and weird?
I've considered running my own IA (https://github.com/internetarchive/heritrix3), which only archives my chrome history, which is stored in a local sqlite db.
I haven't had the time to figure out the details of how the pieces should be glued together.
-
Is there a way to archive groups of webpages similarly to how web archive does it?
You can go all the way and use the same tools as the Internet Archive: https://github.com/internetarchive/heritrix3
- Heritrix: Internet Archive's extensible, web-scale, archival-quality web crawler
- Best Http client for web scraping
-
The Internet Is Rotting
By the way, the technical side of this is very interesting. If you look at the tools mentioned (the wayback machine, but also perma.cc and other archival solutions), almost all of them rely on a single semi-modern tech stack that produces WARCs (web archives - ISO - ISO 28500:2017 https://iipc.github.io/warc-specifications/specifications/wa...).
The main crawler still seems to be heritrix3 (https://github.com/internetarchive/heritrix3), but there's a great little ecosystem with tools such as webrecorder and warcprox.
Still, I've read through the code of these tools and am feeling that they are failing in the face of the modern web with single page apps, mobile phone apps and walled gardens. Even newer iterations with browser automation are getting increasingly throttled and blocked and excluded from walled gardens.
Perhaps the time has come for a coordinated, decentralized but omnipresent approach to archival.
What are some alternatives?
yt-dlp - A feature-rich command-line audio/video downloader
Elasticsearch - Free and Open, Distributed, RESTful Search Engine
FireDM - python open source (Internet Download Manager) with multi-connections, high speed engine, based on python, LibCurl, and youtube_dl https://github.com/firedm/FireDM
brozzler - brozzler - distributed browser-based web crawler
docker-tubeup - Docker container for Tubeup
upvotocracy-ui-ssr - Free speech Reddit clone
youtube-dlc - Command-line program to download various media from YouTube.com and other sites
UCALC - UCALC (Ultimate Calculator) is an advanced Python-based calculator that was my first major Python project.
yark - YouTube archiving made simple.
squid-dl - a massively parallel yt-dlp-based YouTube downloader
yt-dlc - media downloader and library for various sites.