Migrate 5 TB S3 bucket from one AWS account to another

This page summarizes the projects mentioned and recommended in the original post on /r/aws

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • s5cmd

    Parallel S3 and local filesystem execution tool.

    In other threads, I have seen this tool mentioned in another thread for 200 TB and about half a billion files, so your challenge should barely make it break a sweat: https://github.com/peak/s5cmd

  • boto3

    AWS SDK for Python

    Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • aioboto3

    Wrapper to use boto3 resources with the aiobotocore async backend

    Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts