Gcloud storage: up to 94% faster data transfers for Cloud Storage

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • SonarQube - Static code analysis for 29 languages.
  • Scout APM - Truly a developer’s best friend
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • Zigi - Close all those tabs. Zigi will handle your updates.
  • rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    From the article:

    "The new gcloud storage CLI offers significant performance improvements over the existing gsutil ..."

    It makes sense to see this comparison because 'gsutil' was relatively poor - we[1] made a decision to deprecate the 'gsutil' binary[2] in our environment in favor of rclone[3] because rclone is better in every way.

    We made the same decision for 's3cmd'.

    All of that to say: the biggest question to ask about this is "how does it compare to rclone". I suspect the answer is that authentication and token handling (and things like that) are much better as rclone has a pretty clunky workflow for access auth, etc., for google cloud resources. That is not the case for Amazon resources which behave just as you'd expect them to, with regard to API tokens, and make it hard to justify using any tool other than rclone.

    [1] rsync.net

    [2] Yes, a binary - we "freeze" python into binary exe because we don't allow interpreters in our environment.

    [3] https://rclone.org/

  • subpar

    Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

    par- I assume this is the open sourced version https://github.com/google/subpar

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • s5cmd

    Parallel S3 and local filesystem execution tool.

  • s3-parallel-put

    Parallel uploads to Amazon AWS S3

    We often have to sync hundreds of thousands of files to S3 and we use this which often hits 300 files a second on a 1Gbps link:

    https://github.com/mishudark/s3-parallel-put

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts