Gcloud storage: up to 94% faster data transfers for Cloud Storage

1. rclone

   1 979 49,910 9.8 Go

   "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

   

   From the article:

   "The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."

   It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.

   We made the same decision for 's3cmd'.

   All of that to say: the biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky workflow for access auth, etc., for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.

   [1] rsync.net

   [2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.

   [3] https://rclone.org/

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. subpar

   2 2 566 10.0 Python

   Discontinued Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

   

   par- I assume this is the open sourced version https://github.com/google/subpar

4. s5cmd

   3 13 2,996 6.5 Go

   Parallel S3 and local filesystem execution tool.

   

5. s3-parallel-put

   4 1 316 10.0 Python

   Parallel uploads to Amazon AWS S3

   

   We often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link:

   https://github.com/mishudark/s3-parallel-put

Suggest a related project