archivenow
openlibrary
archivenow | openlibrary | |
---|---|---|
4 | 409 | |
391 | 4,848 | |
1.0% | 1.3% | |
3.3 | 9.9 | |
4 months ago | 7 days ago | |
Python | Python | |
MIT License | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
archivenow
-
Best way to feed Wayback Machine a list of URLs?
I crawled a website I want to make sure is completely captured by Wayback Machine but now I need to figure out how to efficiently "feed" all the URLs into Wayback. I found archivenow but I'm terrible at Python so I'm not sure the best way to direct the program at the txt file and preferably create another txt/csv file listing the original url with the new archived url. Any help would be greatly appreciated!
-
Match Thread: West Brom vs Liverpool | Premier League
#!/bin/bash function __longnow(){ # Use: Takes a txt file with one link on each line and pushes all the links to the internet archive # References: # https://unix.stackexchange.com/questions/181254/how-to-use-grep-and-cut-in-script-to-obtain-website-urls-from-an-html-file # https://github.com/oduwsdl/archivenow # For the double underscore, see: https://stackoverflow.com/questions/13797087/bash-why-double-underline-for-private-functions-why-for-bash-complet/15181999 input=$1 counter=1 while IFS= read -r line do wait if [ $(($counter % 15)) -eq 0 ] then printf "\nArchive.org doesn't accept more than 15 links per min; sleeping for 1min...\n" sleep 1m fi echo "Url: $line" archivenow --ia $line >& 1 ## alternatively, archivenow --all $line >& 1 if you want to use all archive services rather than just the internet archive counter=$((counter+1)) done < "$input" } echo 'Gaza' | sed 's/^.*: //' | sed 's/ /%20/g' | sed 's/^/https://news.google.com/rss/search?q=/' | xargs wget --quiet > /dev/null 2>&1 & wait ## This gets news about Gaza from the Google News API/XML endpoint echo "Gaza" | sed 's/^/search?q=/' | sed 's/^/"/;s/$/"/' | xargs xmllint --format 2>/dev/null | grep "title|pubDate|link" | sed 's/.*>(.*)<.*/\1/' | sed '0~3 a\' >> listofnews.txt ## This parses the xml and appends data about each article to a file called "list of news" echo "Gaza" | sed 's/^/search?q=/' | sed 's/^/"/;s/$/"/' | xargs xmllint --format 2>/dev/null | grep "link" | sed 's/.*>(.*)<.*/\1/' > tempforarchiver.txt ## This just gets the links and creates something to be fed to an archiver service. __longnow tempforarchiver.txt rm search?q=Gaza rm tempforarchiver.txt ## Add this to cron with something like ## $ crontab -e ## 30 22 * * * /the/location/of/this/file ### Without the "#" ## This might give you some grief if bash or the archivenow utility can't be found from within the cron instance.
- Archiving the Gaza conflict
- How to easily save web pages to the Internet Archive's Wayback Machine
openlibrary
- Internet Archive: Open Library
-
Ask HN: Anyone looking for contributors for their open source projects
I'd like to make a pitch for Openlibrary.org the free online library from Internet Archive that includes a fulltext search of millions of books.
I've been volunteering with them on and off for several years and it's always a lovely experience. Their backend is python and frontend mostly from python templates and some Vue for librarian stuff.
Every Tuesday they have a call on Zoom that everyone is welcome to join to share what they're working on, ask for help, and generally chat a bit. It's a great time.
Depending on what you're interested in there's a lot to do from helping build import pipelines for more book entries, writing bots to cleanup data, Performance improvements, better documenting public APIs, etc
I'm currently slowly working on a wikidata integration for their authors page. We also could use some help upgrading to Vue 3, mentors for Google summer of code would be helpful, find of ML projects needing help, moving away from old jQuery libraries, etc.
They can be quite responsive to PRs too like I blogged about here: https://blog.rayberger.org/idea-to-merged-in-less-than-30-mi...
For example, here's a small issue that could use some help on the python side: https://github.com/internetarchive/openlibrary/issues/8928
-
Building an Open Source Decentralized E-Book Search Engine
OpenLibrary does provide search access to full texts. For example: https://openlibrary.org/search/inside?q=%22institutional+thi...
It is open source and they're always looking for contributors. I think they'd especially welcome help improving search!
https://github.com/internetarchive/openlibrary/
- Show HN: Mutable.ai – Turn your codebase into a Wiki
-
MLIS books available digitally?
Check out https://openlibrary.org. You can search ´library science’, librarian’, etc, and something should come up. Just select the ‘ebooks’ option to search for items within the collection. And you can narrow the search by subject, etc.
- HMF a “legal” website to download books
-
NaNoWriMo: National Novel Writing Month
Right now I'm in the middle of the chicken and the egg problem where we don't have enough authors cataloging their publications and b/c of that obviously readers are not interested in using the site.
I've gone back and forth with taking Open Libray's [0] catalog as that would at least flesh out our collection of books but then I'd have to deal with verifying authors to accounts so they can access their books. Which sounds like a major headache and also just defeats the concept of building a community.
Since this is really a weekend project, I'm just going to keep building the tools out to perfection and hope people will trickle in over time.
Luckily for me I just want to write, so the tools I'm building are exactly what works for my writing goals and I think overtime others will find the same value.
[0] https://openlibrary.org
-
is there any way to read books for free?
Here's one: https://openlibrary.org/
-
YSK: You can access many old and out of print hiking books from the Internet Archive's Open Library
The Internet Archive runs what they call the Open Library, which is a unique concept on the traditional library. You can sign-up with minimal details and digitally check out many scanned books from libraries all over the world. The only caveat is that almost all of the books are older editions - ones that would be impossible to find locally. It's great if you're looking for old routes, a look back in time, details about obscure areas, or just prefer to read a book rather than browse AllTrails. Please do still support local authors whenever you can as guidebooks take hundreds of hours to create and are slowly going extinct.
-
🐍🐍 23 issues to grow yourself as an exceptional open-source Python expert 🧑💻 🥇
Repo : https://github.com/internetarchive/openlibrary
What are some alternatives?
wayback-machine-spn-scripts - Bash scripts which interact with Internet Archive Wayback Machine's Save Page Now
DeDRM_tools - DeDRM tools for ebooks
videoduplicatefinder - Video Duplicate Finder - Crossplatform
calibre - The official source code repository for the calibre ebook manager
bypass-paywalls-chrome - Bypass Paywalls web browser extension for Chrome and Firefox.
launcher - Launcher for Flashpoint Archive
ArchiveBox - 🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
web - The source code for the Standard Ebooks website.
stylegan2-pytorch - Simplest working implementation of Stylegan2, state of the art generative adversarial network, in Pytorch. Enabling everyone to experience disentanglement
RegExr - RegExr is a HTML/JS based tool for creating, testing, and learning about Regular Expressions.
Chocolatey - Chocolatey - the package manager for Windows
savepagenow - A simple Python wrapper and command-line interface for archive.org’s "Save Page Now" capturing service