Go S3

Open-source Go projects categorized as S3 | Edit details
Related topics: #Golang #Go #AWS #Storage #Filesystem

Top 23 Go S3 Projects

  • GitHub repo minio

    High Performance, Kubernetes Native Object Storage

    Project mention: Refactoring #4: Using Minio to work with S3 buckets locally | dev.to | 2022-01-09

    Minio is an open-source object storage server with an Amazon S3 compatible API. If you’re using Homestead as your working environment, you’re super lucky; Minio is pretty easy to install, barely an inconvenience. Follow the instructions on the Laravel docs above, and you should be ready for the next step.

  • GitHub repo rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    Project mention: Dual booting now and planning to fully convert with 22.04 LTS release and I have a couple questions | reddit.com/r/Ubuntu | 2022-01-22

    * Via 'rclone' - https://rclone.org/ - one way sync client, open source and free

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • GitHub repo Seaweed File System

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    Project mention: Updated MinIO NVMe Benchmarks: 2.6Tpbs on Get and 1.6 on Put | news.ycombinator.com | 2022-01-04

    This is a fair complaint. :)

    For filer metadata, you should just pick the one you are most familiar with.

    There is a wiki page for production setup. https://github.com/chrislusf/seaweedfs/wiki/Production-Setup

  • GitHub repo thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    Project mention: Do i really want to mess with OpenTelemetry, or just hook straight into Datadog | reddit.com/r/devops | 2022-01-18

    Old company was on a smaller scale than the new one. But still https://thanos.io/

  • GitHub repo Filestash

    🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...

    Project mention: Fly.io – Free Postgres Databases (and free storage volumes, up to 3GB total) | news.ycombinator.com | 2022-01-20

    > If we can sift through 1,000 free users and find one who converts to $25k per month, we're delighted.

    Does this ever happen? I was contacted by a F500 who needed some custom version of my open source project (https://github.com/mickael-kerjean/filestash). I'm aiming 2k a month with support and I'm afraid discussion won't move on if it's considered too expensive already. 25k sounds unreal

  • GitHub repo juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: JuiceFS 1.0-beta1 was released under Apache 2.0 | reddit.com/r/golang | 2022-01-13
  • GitHub repo goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    Project mention: What you gonna add to your selfhost stack this year? | reddit.com/r/selfhosted | 2022-01-02

    will probably experiment with https://github.com/kahing/goofys and https://litestream.io/ to make services more easily moved between the devices :) Also, will continue working on https://synpse.net/ to make the operations easier.

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • GitHub repo SFTPGo

    Fully featured and highly configurable SFTP server with optional HTTP, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    Project mention: Remote Backup Solution | reddit.com/r/selfhosted | 2022-01-16

    You could also check out SFTPGo

  • GitHub repo chubaofs

    ChubaoFS (abbrev. CBFS) is a cloud native distributed file system and object store.

    Project mention: How are you managing NFS in K8s world? | reddit.com/r/kubernetes | 2021-07-15

    A few weeks ago I took a peek at chubaofs and frankly didn't understand how to install it; but it claims to support RWX.

  • GitHub repo wal-g

    Archival and Restoration for Postgres

    Project mention: Optimizing Postgres Queries at Scale | news.ycombinator.com | 2021-12-28
  • GitHub repo mc

    MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.

  • GitHub repo minio-go

    MinIO Client SDK for Go

    Project mention: Why I cannot access minio via sdk? | reddit.com/r/kubernetes | 2021-11-09

    I have installed minio on my cluster. I am making an API call with go to create a bucket. But I don't why is not responding as in here I did the same this example but I am getting the following response:

  • GitHub repo s5cmd

    Parallel S3 and local filesystem execution tool.

    Project mention: S3/100gbps question | reddit.com/r/aws | 2021-12-05

    I like to use https://github.com/peak/s5cmd

  • GitHub repo s3-benchmark

    Measure Amazon S3's performance from any location.

  • GitHub repo clickhouse-backup

    Tool for easy ClickHouse backup and restore with cloud storages support

    Project mention: Backing up Plausible Analytics database | reddit.com/r/Clickhouse | 2021-08-03

    I set up Plausible Analytics in my Kubernetes cluster and am trying to figure out how to properly back up and restore the Clickhouse database. I am trying to use https://github.com/AlexAkulov/clickhouse-backup but it only supports table of the MergeTree family. Plausible uses a table named schema_migrations which is of type TinyLog, so it's skipped during backups, making restores useless (because the table is restored empty, when Plausible starts it will try to run all the migrations which will fail because the other tables already exist).

  • GitHub repo mort

    Storage and image processing server written in Go

  • GitHub repo helm-s3

    Helm plugin that allows to set up a chart repository in AWS S3.

    Project mention: How do you store your helm charts? | reddit.com/r/devops | 2021-07-22

    we store helm chart sources in GitHub and Jenkins pushes them to S3 using helm s3 plugin. great as we don't need a service like chartmuseum

  • GitHub repo csi-s3

    A Container Storage Interface for S3

    Project mention: How to scale nginx pod when pod is mounting a volume | reddit.com/r/kubernetes | 2021-08-29

    You could also use the s3fs CSI for your storage. There may be some learning curve to getting it working. My only word of advise is to use the examples in the repo, the README.md is stale. I made some notes here.

  • GitHub repo storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

    Project mention: 2021-16 Weekly Report for go-storage: An application-oriented unified storage layer for Golang. | reddit.com/r/golang | 2021-04-23

    More and much easier services support. We have built a code generator to generate code like optional function options. Like go-service-example, we can implement a new service in minutes. For now, we have implemented 10 services and more services are in our roadmap.

  • GitHub repo afs

    Abstract File Storage

  • GitHub repo rpCheckup

    rpCheckup is an AWS resource policy security checkup tool that identifies public, external account access, intra-org account access, and private resources.

    Project mention: AWS GitHub script thoughts | reddit.com/r/hacking | 2021-07-24

    What do people think of this? https://github.com/goldfiglabs/rpCheckup I am in no way affiliated with this - frankly I am worried if i use this ^ it might have other access things in my account? But i think all it needs is read only?

  • GitHub repo s3fs

    S3 FileSystem (fs.FS) implementation (by jszwec)

    Project mention: AWS S3 FileSystem (io/fs) Implementation for Go1.16 | reddit.com/r/golang | 2021-02-01
  • GitHub repo s3-proxy

    S3 Reverse Proxy with GET, PUT and DELETE methods and authentication (OpenID Connect and Basic Auth)

    Project mention: Help configuring an S3 website to be accessible from on prem only | reddit.com/r/aws | 2021-09-10

    Why don't you use just an s3 proxy software like this one? https://oxyno-zeta.github.io/s3-proxy/ There are others

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-01-22.

Go S3 related posts


What are some of the best open-source S3 projects in Go? This list will help you:

Project Stars
1 minio 31,137
2 rclone 31,002
3 Seaweed File System 13,701
4 thanos 9,935
5 Filestash 5,405
6 juicefs 4,715
7 goofys 3,974
8 SFTPGo 3,679
9 chubaofs 2,507
10 wal-g 2,015
11 mc 1,984
12 minio-go 1,494
13 s5cmd 901
14 s3-benchmark 587
15 clickhouse-backup 570
16 mort 443
17 helm-s3 421
18 csi-s3 400
19 storage 295
20 afs 172
21 rpCheckup 139
22 s3fs 122
23 s3-proxy 76
Find remote jobs at our new job board 99remotejobs.com. There are 29 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
Less time debugging, more time building
Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.