Top 23 Go S3 Projects
-
Minio is an open-source object storage server with an Amazon S3 compatible API. If you’re using Homestead as your working environment, you’re super lucky; Minio is pretty easy to install, barely an inconvenience. Follow the instructions on the Laravel docs above, and you should be ready for the next step.
-
rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Project mention: Dual booting now and planning to fully convert with 22.04 LTS release and I have a couple questions | reddit.com/r/Ubuntu | 2022-01-22* Via 'rclone' - https://rclone.org/ - one way sync client, open source and free
-
OPS
OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.
-
Seaweed File System
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
Project mention: Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put | news.ycombinator.com | 2022-01-04This is a fair complaint. :)
For filer metadata, you should just pick the one you are most familiar with.
There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup
-
thanos
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
Project mention: Do i really want to mess with OpenTelemetry, or just hook straight into Datadog | reddit.com/r/devops | 2022-01-18Old company was on a smaller scale than the new one. But still https://thanos.io/
-
Filestash
🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...
Project mention: Fly.io – Free Postgres Databases (and free storage volumes, up to 3GB total) | news.ycombinator.com | 2022-01-20> If we can sift through 1,000 free users and find one who converts to $25k per month, we're delighted.
Does this ever happen? I was contacted by a F500 who needed some custom version of my open source project (https://github.com/mickael-kerjean/filestash). I'm aiming 2k a month with support and I'm afraid discussion won't move on if it's considered too expensive already. 25k sounds unreal
-
-
Project mention: What you gonna add to your selfhost stack this year? | reddit.com/r/selfhosted | 2022-01-02
will probably experiment with https://github.com/kahing/goofys and https://litestream.io/ to make services more easily moved between the devices :) Also, will continue working on https://synpse.net/ to make the operations easier.
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
SFTPGo
Fully featured and highly configurable SFTP server with optional HTTP, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob
You could also check out SFTPGo
-
A few weeks ago I took a peek at chubaofs and frankly didn't understand how to install it; but it claims to support RWX.
-
-
mc
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
-
I have installed minio on my cluster. I am making an API call with go to create a bucket. But I don't why is not responding as in here I did the same this example but I am getting the following response:
-
I like to use https://github.com/peak/s5cmd
-
-
I set up Plausible Analytics in my Kubernetes cluster and am trying to figure out how to properly back up and restore the Clickhouse database. I am trying to use https://github.com/AlexAkulov/clickhouse-backup but it only supports table of the MergeTree family. Plausible uses a table named schema_migrations which is of type TinyLog, so it's skipped during backups, making restores useless (because the table is restored empty, when Plausible starts it will try to run all the migrations which will fail because the other tables already exist).
-
-
we store helm chart sources in GitHub and Jenkins pushes them to S3 using helm s3 plugin. great as we don't need a service like chartmuseum
-
Project mention: How to scale nginx pod when pod is mounting a volume | reddit.com/r/kubernetes | 2021-08-29
You could also use the s3fs CSI for your storage. There may be some learning curve to getting it working. My only word of advise is to use the examples in the repo, the README.md is stale. I made some notes here.
-
Project mention: 2021-16 Weekly Report for go-storage: An application-oriented unified storage layer for Golang. | reddit.com/r/golang | 2021-04-23
More and much easier services support. We have built a code generator to generate code like optional function options. Like go-service-example, we can implement a new service in minutes. For now, we have implemented 10 services and more services are in our roadmap.
-
-
rpCheckup
rpCheckup is an AWS resource policy security checkup tool that identifies public, external account access, intra-org account access, and private resources.
What do people think of this? https://github.com/goldfiglabs/rpCheckup I am in no way affiliated with this - frankly I am worried if i use this ^ it might have other access things in my account? But i think all it needs is read only?
-
Project mention: AWS S3 FileSystem (io/fs) Implementation for Go1.16 | reddit.com/r/golang | 2021-02-01
-
s3-proxy
S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)
Project mention: Help configuring an S3 website to be accessible from on prem only | reddit.com/r/aws | 2021-09-10Why don't you use just an s3 proxy software like this one? https://oxyno-zeta.github.io/s3-proxy/ There are others
Go S3 related posts
- Dual booting now and planning to fully convert with 22.04 LTS release and I have a couple questions
- Got a new Surface Pro 8 and I've some questions (ex MacOS user)
- rclone does not resume via sftp?
- Weekly Coders, Hackers & All Tech related thread - 20/01/2022
- 50TB of storage, no sync needed
- Getting Rclone failed Error on Sidenoder
- What is the fastest way to transfer 100s of GB from one VM to another VM?
Index
What are some of the best open-source S3 projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | minio | 31,137 |
| 2 | rclone | 31,002 |
| 3 | Seaweed File System | 13,701 |
| 4 | thanos | 9,935 |
| 5 | Filestash | 5,405 |
| 6 | juicefs | 4,715 |
| 7 | goofys | 3,974 |
| 8 | SFTPGo | 3,679 |
| 9 | chubaofs | 2,507 |
| 10 | wal-g | 2,015 |
| 11 | mc | 1,984 |
| 12 | minio-go | 1,494 |
| 13 | s5cmd | 901 |
| 14 | s3-benchmark | 587 |
| 15 | clickhouse-backup | 570 |
| 16 | mort | 443 |
| 17 | helm-s3 | 421 |
| 18 | csi-s3 | 400 |
| 19 | storage | 295 |
| 20 | afs | 172 |
| 21 | rpCheckup | 139 |
| 22 | s3fs | 122 |
| 23 | s3-proxy | 76 |
Are you hiring? Post a new remote job listing for free.
