Gcloud storage: up to 94% faster data transfers for Cloud Storage

• rclone

  963 43,720 9.8 Go

  "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

  

From the article:

"The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."

It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.

We made the same decision for 's3cmd'.

All of that to say: the biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky workflow for access auth, etc., for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.

[1] rsync.net

[2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.

[3] https://rclone.org/

• subpar

  2 566 10.0 Python

  Discontinued Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

  

par- I assume this is the open sourced version https://github.com/google/subpar

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• s5cmd

  11 2,324 7.3 Go

  Parallel S3 and local filesystem execution tool.

  

• s3-parallel-put

  1 312 10.0 Python

  Parallel uploads to Amazon AWS S3

We often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link:

https://github.com/mishudark/s3-parallel-put

Suggest a related project