Looking for some help in downloading a few thousand files from archive.org on ubuntu. wget is estimated to take 2 months... I figured I should ask the fellow data-hoarders!

• iadownloader

  1 4 10.0 Python

  Auto-download files and collections from Internet Archive

I've used this [https://github.com/rsvensson/iadownloader] for a similar use case, but you might also consider one of these...

• archive-downloader

  1 1 1.8 Python

  A downloader for archive.org

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• BaseCase-3

  2 2 0.0 Python

  Discontinued This is a Python Application that can be used to gather all files of a certain type from any archive.com repository

• internetarchive-downloader

  7 119 3.6 Python

  Simultaneous, resumable and hash-verified downloads from Internet Archive (archive.org)

• internetarchive

  17 1,513 8.3 Python

  A Python and Command-Line Interface to Archive.org

• GGet

  1 15 10.0 Go

  Multithreaded download accelerator written in Go

This sounds pretty easy to do with Go, spawn hundreds of threads (which have a ridiculously small ram footprint) and off you go. A google search yielded this: https://github.com/ashwinGokhale/GGet

Suggest a related project