Top 11 Go Cloud Storage Projects
-
rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Project mention: Are files safe from being deleted in ProtonDrive beta? | reddit.com/r/ProtonMail | 2021-12-01I can recommend using rclone with backblaze B2, where you can encrypt data before being uploaded to the B2 storage bucket. It's a little work getting rclone correctly setup with the encryption, but once done you just need to kick off a "backup script" to upload data.
-
Cloudreve
🌩支持多家云存储的云盘系统 (Self-deployed file management and sharing system, supports multiple storage providers)
Project mention: [File-Sharing] Does anyone heard of CloudReve ? | reddit.com/r/selfhosted | 2021-08-18 -
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
> We've had some ideas around using this for distributed querying: in our case, each node responsible for a given partition of a dataset would be able to download just the objects in that partition on the fly (though constraint pruning), so we wouldn't need to knowingly seed each worker with data.
IMHO, if you're going to do this, I'd recommend not doing this in Postgres itself, but rather doing it at the filesystem level. It's effectively just a tiered-storage read-through cache, and filesystems have those all figured out already.
You know how pgBackRest does "partial restore" (https://pgbackrest.org/user-guide.html#restore/option-db-inc...), by making all the heap files seem to be there, but actually they're empty sparse files that just happen to have the right allocated length to make PG happy?
Imagine taking one of the object-storage FUSE filesystems, e.g. https://github.com/kahing/goofys, and modding it so that it represents all not-yet-fetched files under readdir(2) with an equivalent representation.
Then just make your pg_base dir an overlayfs mount for:
• top layer: tmpfs (only necessary if you don't give temp tables their own tablespace)
-
SFTPGo
Fully featured and highly configurable SFTP server with optional HTTP, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob
Project mention: SFTPGo 2.2.0 released with built-in 2FA, plugin support and more | reddit.com/r/linux | 2021-11-27 -
Project mention: Check Certificate Expiration date from Bash to Go - help needed | reddit.com/r/golang | 2021-11-21
I'm struggling to find the idiomatic Go SDK API from GCP and how to use the existing one https://cloud.google.com/go/docs/reference # don't have DNS package https://github.com/googleapis/google-cloud-go # don't have DNS package https://github.com/googleapis/google-api-go-client # has DNS package but I can't find one way to list my managed DNS zones So in this case, what can I do? So far I've done the following, I want to change the func hostList() to something dynamic generated based on my Google Cloud DNS zones.
-
Project mention: Daptin - Backend As A Service - GraphQL/JSON-API Headless CMS in Golang | reddit.com/r/golang | 2021-06-15
https://daptin.github.io/daptin/#guides
-
gokv
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
-
-
Project mention: 2021-16 Weekly Report for go-storage: An application-oriented unified storage layer for Golang. | reddit.com/r/golang | 2021-04-23
More and much easier services support. We have built a code generator to generate code like optional function options. Like go-service-example, we can implement a new service in minutes. For now, we have implemented 10 services and more services are in our roadmap.
-
zfsbackup-go
Backup ZFS snapshots to cloud storage such as Google, Amazon, Azure, etc. Built with the enterprise in mind.
Project mention: Incremental backups / replication to non-ZFS destinations? | reddit.com/r/zfs | 2021-09-20Z3 doesn't seem to be maintained. This script seems to be a oneoff. ZFSBackup Go seems more full featured but also unmaintained and out of date as well.
-
Project mention: Elastic Harp v0.2.1 - Secret management pipeline toolchain | reddit.com/r/devsecops | 2021-11-18
-
The advantage of that above just a simple "these tests are allowed to fail" is that, after adding a feature or fixing a bug, the developer is forced to look into the tests. If tests start to pass, the only job the developer has to do, is to remove them from the expected-to-fail list, and what a joy is that 🎉. From that point on the test has to pass in all future runs, and we are sure not to introduce any regressions. If we would only have an allowed-to-fail list, there would be no pressure to remove tests from that list, humans are humans, so they would forget or miss some. Potentially a bug could get fixed, then see some regression again and none of that would be noticed by the test-suite. So let the computers do what they are good at - automate!
Go Cloud Storage related posts
- Using Google Drive desktop app, can I upload files via File Explorer and opt to not store them locally?
- Best Cloud Backup Solution for systems Images?
- poof! Safe, secure, and forensics-resistant backup - request for comments
- Filesystem question - looking for a remote FS that can be accessed purely from client side
- I am A current windows 11 user and i want to Change my main OS to Fedora Linux and i want to know how i can backup everything to onedrive and will they be available when i run windows 11 on a Virtual Machine?
- What can I do if I want to store around 50-64tbs and have a budget of 80 euros monthly?
- Backup Setup
Index
What are some of the best open-source Cloud Storage projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | rclone | 29,817 |
| 2 | Cloudreve | 12,098 |
| 3 | goofys | 3,900 |
| 4 | SFTPGo | 3,355 |
| 5 | google-cloud | 2,708 |
| 6 | daptin | 1,479 |
| 7 | gokv | 376 |
| 8 | storage | 274 |
| 9 | zfsbackup-go | 221 |
| 10 | harp | 102 |
| 11 | reva | 82 |
Are you hiring? Post a new remote job listing for free.
