Our great sponsors
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
In other threads, I have seen this tool mentioned in another thread for 200 TB and about half a billion files, so your challenge should barely make it break a sweat: https://github.com/peak/s5cmd
Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.
Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.