archivenow VS videoduplicatefinder

I crawled a website I want to make sure is completely captured by Wayback Machine but now I need to figure out how to efficiently "feed" all the URLs into Wayback. I found archivenow but I'm terrible at Python so I'm not sure the best way to direct the program at the txt file and preferably create another txt/csv file listing the original url with the new archived url. Any help would be greatly appreciated!

#!/bin/bash function __longnow(){ # Use: Takes a txt file with one link on each line and pushes all the links to the internet archive # References: # https://unix.stackexchange.com/questions/181254/how-to-use-grep-and-cut-in-script-to-obtain-website-urls-from-an-html-file # https://github.com/oduwsdl/archivenow # For the double underscore, see: https://stackoverflow.com/questions/13797087/bash-why-double-underline-for-private-functions-why-for-bash-complet/15181999 input=$1 counter=1 while IFS= read -r line do wait if [ $(($counter % 15)) -eq 0 ] then printf "\nArchive.org doesn't accept more than 15 links per min; sleeping for 1min...\n" sleep 1m fi echo "Url: $line" archivenow --ia $line >& 1 ## alternatively, archivenow --all $line >& 1 if you want to use all archive services rather than just the internet archive counter=$((counter+1)) done < "$input" } echo 'Gaza' | sed 's/^.*: //' | sed 's/ /%20/g' | sed 's/^/https://news.google.com/rss/search?q=/' | xargs wget --quiet > /dev/null 2>&1 & wait ## This gets news about Gaza from the Google News API/XML endpoint echo "Gaza" | sed 's/^/search?q=/' | sed 's/^/"/;s/$/"/' | xargs xmllint --format 2>/dev/null | grep "title|pubDate|link" | sed 's/.*>(.*)<.*/\1/' | sed '0~3 a\' >> listofnews.txt ## This parses the xml and appends data about each article to a file called "list of news" echo "Gaza" | sed 's/^/search?q=/' | sed 's/^/"/;s/$/"/' | xargs xmllint --format 2>/dev/null | grep "link" | sed 's/.*>(.*)<.*/\1/' > tempforarchiver.txt ## This just gets the links and creates something to be fed to an archiver service. __longnow tempforarchiver.txt rm search?q=Gaza rm tempforarchiver.txt ## Add this to cron with something like ## $ crontab -e ## 30 22 * * * /the/location/of/this/file ### Without the "#" ## This might give you some grief if bash or the archivenow utility can't be found from within the cron instance.

Step 1: download the daily build, which in my case, would be App-osx-arm64.tar.gz since I have an M1. I just unzipped it to /Downloads/VDF

I've been trying for a bit now to get Video Duplicate Finder from GitHub to run without success.

Ooh this looks nice. And might replace at least 2 tools. For video comparison I currently use Video Duplicate Finder which has a few features this doesn't. Under settings it lets you check Ignore White Pixels and Ignore Black Pixels which helps lower false positives on videos with lots of white/black backgrounds or keyframes. There's an option to compare horizontally flipped videos, and a few more that are a bit more esoteric for me to understand. But it might be worth peeking at how they accomplish some of it.

Here's the link: https://github.com/0x90d/videoduplicatefinder. The directions to install seem minimal. Please explain for someone who isn't familiar with terminal.

I have used this with great success https://github.com/0x90d/videoduplicatefinder

I've used Video Duplicate Finder for this before. (Yes, it works on images, supposedly). It's pretty buggy but it does the job.

I use Video Duplicate Finder in a docker container on my media server. It shows you at a glance which version of a file has the longest duration, highest frame rate, bitrate, resolution, file size, etc.

For videos though I used something different. https://github.com/0x90d/videoduplicatefinder - its not quite as polished as some software but the results more than make up for it.