Top 11 Go Cloud Storage Projects
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex FilesProject mention: Are files safe from being deleted in ProtonDrive beta? | reddit.com/r/ProtonMail | 2021-12-01
I can recommend using rclone with backblaze B2, where you can encrypt data before being uploaded to the B2 storage bucket. It's a little work getting rclone correctly setup with the encryption, but once done you just need to kick off a "backup script" to upload data.
🌩支持多家云存储的云盘系统 (Self-deployed file management and sharing system, supports multiple storage providers)Project mention: [File-Sharing] Does anyone heard of CloudReve ? | reddit.com/r/selfhosted | 2021-08-18
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
a high-performance, POSIX-ish Amazon S3 file system written in GoProject mention: Scalable PostgreSQL Connection Pooler | news.ycombinator.com | 2021-11-12
> We've had some ideas around using this for distributed querying: in our case, each node responsible for a given partition of a dataset would be able to download just the objects in that partition on the fly (though constraint pruning), so we wouldn't need to knowingly seed each worker with data.
IMHO, if you're going to do this, I'd recommend not doing this in Postgres itself, but rather doing it at the filesystem level. It's effectively just a tiered-storage read-through cache, and filesystems have those all figured out already.
You know how pgBackRest does "partial restore" (https://pgbackrest.org/user-guide.html#restore/option-db-inc...), by making all the heap files seem to be there, but actually they're empty sparse files that just happen to have the right allocated length to make PG happy?
Imagine taking one of the object-storage FUSE filesystems, e.g. https://github.com/kahing/goofys, and modding it so that it represents all not-yet-fetched files under readdir(2) with an equivalent representation.
Then just make your pg_base dir an overlayfs mount for:
• top layer: tmpfs (only necessary if you don't give temp tables their own tablespace)
Fully featured and highly configurable SFTP server with optional HTTP, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure BlobProject mention: SFTPGo 2.2.0 released with built-in 2FA, plugin support and more | reddit.com/r/linux | 2021-11-27
Google Cloud Client Libraries for Go.Project mention: Check Certificate Expiration date from Bash to Go - help needed | reddit.com/r/golang | 2021-11-21
I'm struggling to find the idiomatic Go SDK API from GCP and how to use the existing one https://cloud.google.com/go/docs/reference # don't have DNS package https://github.com/googleapis/google-cloud-go # don't have DNS package https://github.com/googleapis/google-api-go-client # has DNS package but I can't find one way to list my managed DNS zones So in this case, what can I do? So far I've done the following, I want to change the func hostList() to something dynamic generated based on my Google Cloud DNS zones.
Daptin - Backend As A Service - GraphQL/JSON-API Headless CMSProject mention: Daptin - Backend As A Service - GraphQL/JSON-API Headless CMS in Golang | reddit.com/r/golang | 2021-06-15
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
Run Linux Software Faster and Safer than Linux with Unikernels.
A vendor-neutral storage library for Golang: Write once, run on every storage service.Project mention: 2021-16 Weekly Report for go-storage: An application-oriented unified storage layer for Golang. | reddit.com/r/golang | 2021-04-23
More and much easier services support. We have built a code generator to generate code like optional function options. Like go-service-example, we can implement a new service in minutes. For now, we have implemented 10 services and more services are in our roadmap.
Backup ZFS snapshots to cloud storage such as Google, Amazon, Azure, etc. Built with the enterprise in mind.Project mention: Incremental backups / replication to non-ZFS destinations? | reddit.com/r/zfs | 2021-09-20
Z3 doesn't seem to be maintained. This script seems to be a oneoff. ZFSBackup Go seems more full featured but also unmaintained and out of date as well.
Secret management toolchain (by elastic)Project mention: Elastic Harp v0.2.1 - Secret management pipeline toolchain | reddit.com/r/devsecops | 2021-11-18
IOP (Inter Operability Platform)Project mention: BDD on Software Rewrite | dev.to | 2021-02-15
The advantage of that above just a simple "these tests are allowed to fail" is that, after adding a feature or fixing a bug, the developer is forced to look into the tests. If tests start to pass, the only job the developer has to do, is to remove them from the expected-to-fail list, and what a joy is that 🎉. From that point on the test has to pass in all future runs, and we are sure not to introduce any regressions. If we would only have an allowed-to-fail list, there would be no pressure to remove tests from that list, humans are humans, so they would forget or miss some. Potentially a bug could get fixed, then see some regression again and none of that would be noticed by the test-suite. So let the computers do what they are good at - automate!
Go Cloud Storage related posts
Using Google Drive desktop app, can I upload files via File Explorer and opt to not store them locally?
1 project | reddit.com/r/gsuite | 30 Nov 2021
Best Cloud Backup Solution for systems Images?
1 project | reddit.com/r/sysadmin | 29 Nov 2021
poof! Safe, secure, and forensics-resistant backup - request for comments
3 projects | reddit.com/r/DataHoarder | 29 Nov 2021
Filesystem question - looking for a remote FS that can be accessed purely from client side
1 project | reddit.com/r/linux | 29 Nov 2021
I am A current windows 11 user and i want to Change my main OS to Fedora Linux and i want to know how i can backup everything to onedrive and will they be available when i run windows 11 on a Virtual Machine?
2 projects | reddit.com/r/onedrive | 27 Nov 2021
What can I do if I want to store around 50-64tbs and have a budget of 80 euros monthly?
1 project | reddit.com/r/DataHoarder | 26 Nov 2021
2 projects | reddit.com/r/Arqbackup | 24 Nov 2021
What are some of the best open-source Cloud Storage projects in Go? This list will help you:
Are you hiring? Post a new remote job listing for free.