Go Cloud Storage

Open-source Go projects categorized as Cloud Storage | Edit details
Related topics: #Golang #S3 #Go #Cloud #azure-blob

Top 11 Go Cloud Storage Projects

  • GitHub repo rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    Project mention: Are files safe from being deleted in ProtonDrive beta? | reddit.com/r/ProtonMail | 2021-12-01

    I can recommend using rclone with backblaze B2, where you can encrypt data before being uploaded to the B2 storage bucket. It's a little work getting rclone correctly setup with the encryption, but once done you just need to kick off a "backup script" to upload data.

  • GitHub repo Cloudreve

    🌩支持多家云存储的云盘系统 (Self-deployed file management and sharing system, supports multiple storage providers)

    Project mention: [File-Sharing] Does anyone heard of CloudReve ? | reddit.com/r/selfhosted | 2021-08-18
  • Scout APM

    Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

  • GitHub repo goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    Project mention: Scalable PostgreSQL Connection Pooler | news.ycombinator.com | 2021-11-12

    > We've had some ideas around using this for distributed querying: in our case, each node responsible for a given partition of a dataset would be able to download just the objects in that partition on the fly (though constraint pruning), so we wouldn't need to knowingly seed each worker with data.

    IMHO, if you're going to do this, I'd recommend not doing this in Postgres itself, but rather doing it at the filesystem level. It's effectively just a tiered-storage read-through cache, and filesystems have those all figured out already.

    You know how pgBackRest does "partial restore" (https://pgbackrest.org/user-guide.html#restore/option-db-inc...), by making all the heap files seem to be there, but actually they're empty sparse files that just happen to have the right allocated length to make PG happy?

    Imagine taking one of the object-storage FUSE filesystems, e.g. https://github.com/kahing/goofys, and modding it so that it represents all not-yet-fetched files under readdir(2) with an equivalent representation.

    Then just make your pg_base dir an overlayfs mount for:

    • top layer: tmpfs (only necessary if you don't give temp tables their own tablespace)

  • GitHub repo SFTPGo

    Fully featured and highly configurable SFTP server with optional HTTP, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    Project mention: SFTPGo 2.2.0 released with built-in 2FA, plugin support and more | reddit.com/r/linux | 2021-11-27
  • GitHub repo google-cloud

    Google Cloud Client Libraries for Go.

    Project mention: Check Certificate Expiration date from Bash to Go - help needed | reddit.com/r/golang | 2021-11-21

    I'm struggling to find the idiomatic Go SDK API from GCP and how to use the existing one https://cloud.google.com/go/docs/reference # don't have DNS package https://github.com/googleapis/google-cloud-go # don't have DNS package https://github.com/googleapis/google-api-go-client # has DNS package but I can't find one way to list my managed DNS zones So in this case, what can I do? So far I've done the following, I want to change the func hostList() to something dynamic generated based on my Google Cloud DNS zones.

  • GitHub repo daptin

    Daptin - Backend As A Service - GraphQL/JSON-API Headless CMS

    Project mention: Daptin - Backend As A Service - GraphQL/JSON-API Headless CMS in Golang | reddit.com/r/golang | 2021-06-15


  • GitHub repo gokv

    Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

  • Nanos

    Run Linux Software Faster and Safer than Linux with Unikernels.

  • GitHub repo storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

    Project mention: 2021-16 Weekly Report for go-storage: An application-oriented unified storage layer for Golang. | reddit.com/r/golang | 2021-04-23

    More and much easier services support. We have built a code generator to generate code like optional function options. Like go-service-example, we can implement a new service in minutes. For now, we have implemented 10 services and more services are in our roadmap.

  • GitHub repo zfsbackup-go

    Backup ZFS snapshots to cloud storage such as Google, Amazon, Azure, etc. Built with the enterprise in mind.

    Project mention: Incremental backups / replication to non-ZFS destinations? | reddit.com/r/zfs | 2021-09-20

    Z3 doesn't seem to be maintained. This script seems to be a oneoff. ZFSBackup Go seems more full featured but also unmaintained and out of date as well.

  • GitHub repo harp

    Secret management toolchain (by elastic)

    Project mention: Elastic Harp v0.2.1 - Secret management pipeline toolchain | reddit.com/r/devsecops | 2021-11-18
  • GitHub repo reva

    IOP (Inter Operability Platform)

    Project mention: BDD on Software Rewrite | dev.to | 2021-02-15

    The advantage of that above just a simple "these tests are allowed to fail" is that, after adding a feature or fixing a bug, the developer is forced to look into the tests. If tests start to pass, the only job the developer has to do, is to remove them from the expected-to-fail list, and what a joy is that 🎉. From that point on the test has to pass in all future runs, and we are sure not to introduce any regressions. If we would only have an allowed-to-fail list, there would be no pressure to remove tests from that list, humans are humans, so they would forget or miss some. Potentially a bug could get fixed, then see some regression again and none of that would be noticed by the test-suite. So let the computers do what they are good at - automate!

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-12-01.

Go Cloud Storage related posts


What are some of the best open-source Cloud Storage projects in Go? This list will help you:

Project Stars
1 rclone 29,817
2 Cloudreve 12,098
3 goofys 3,900
4 SFTPGo 3,355
5 google-cloud 2,708
6 daptin 1,479
7 gokv 376
8 storage 274
9 zfsbackup-go 221
10 harp 102
11 reva 82
Find remote jobs at our new job board 99remotejobs.com. There are 32 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives