Go S3

Open-source Go projects categorized as S3

Top 23 Go S3 Projects

  • minio

    The Object Store for AI Data Infrastructure

    Project mention: A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes | news.ycombinator.com | 2024-02-22

    Looks like minio added this in 2022:

    https://github.com/minio/minio/pull/15433

  • rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    Project mention: Rclone syncs your files to cloud storage | news.ycombinator.com | 2024-01-26

    Says that Apple doesn't provide a multi platform API. It doesn't provide any official supported way to access iCloud from Windows, Linux.

    There's a ticket covering everything you might ever want to know:

    https://github.com/rclone/rclone/issues/1778

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    Project mention: Billion File Filesystem | news.ycombinator.com | 2024-02-08

    If you want/need to take out the metadata, there's some nice solutions for that https://github.com/seaweedfs/seaweedfs

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10

    Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

  • litestream

    Streaming replication for SQLite.

    Project mention: SQLite3 Replication: A Wizard's Guide🧙🏽 | dev.to | 2024-02-27

    This post intends to help you setup replication for SQLite using Litestream.

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

    Support for Kerberos keytab files

  • SFTPGo

    Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    Project mention: What you guys are hosting instead of Nextcloud? I'm sick of it. | /r/selfhosted | 2023-11-29

    EDIT: Thanks for the recommendations from all of you!! I've chose to use the below: - Files: sftpgo - Calendar: baikal - Notes: memos (But beware, it sends opt-out telemetry) - Network folder: webdav on sftpgo

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    Project mention: Is Posix Outdated? | news.ycombinator.com | 2023-10-19

    The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

    Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.

    So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

    Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

    That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

    https://github.com/s3fs-fuse/s3fs-fuse

    https://github.com/kahing/goofys

    https://www.cuno.io/

  • wal-g

    Archival and Restoration for databases in the Cloud

    Project mention: Introducing Read Replicas | dev.to | 2023-12-15

    Streaming replication minimizes replication lag, while file-based log shipping provides a fallback. For file-based log shipping, we use our existing Point In Time Recovery (PITR) infrastructure. We regularly archive files from the primary using WAL-G, an open source archival and restoration tool, and ship the WAL files to S3.

  • mc

    Simple | Fast tool to manage MinIO clusters :cloud:

  • S3Scanner

    Scan for misconfigured S3 buckets across S3-compatible APIs!

  • s5cmd

    Parallel S3 and local filesystem execution tool.

    Project mention: GitHub issues from top Open Source Golang Repositories that you should contribute to | dev.to | 2024-01-15

    s5cmd - Extended character support for s3 compatible backend

  • minio-go

    MinIO Go client SDK for S3 compatible object storage

    Project mention: Need to stream large files to S3 using Go Fiber/Fasthttp | /r/golang | 2023-04-27
  • docker-volume-backup

    Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage

    Project mention: I have a question about | /r/paulsdevblog | 2023-11-29

    I am interested in coming up with a backup plan before I get too invested in this setup. I found the docker-volume-backup project that looks like it might be a possible solution. However I'm not sure how to implement it using docker swarm since I am new to all of this. I would be interested in learning what backup solution you use for your docker swarm servers.

  • rill

    Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code. (by rilldata)

    Project mention: Governments on GitHub | news.ycombinator.com | 2023-06-09
  • clickhouse-backup

    Tool for easy ClickHouse backup and restore using object storage for backup files.

  • s3-benchmark

    Measure Amazon S3's performance from any location.

    Project mention: S3 Benchmark | news.ycombinator.com | 2023-06-17
  • csi-s3

    A Container Storage Interface for S3

  • helm-s3

    ⎈ Helm plugin that adds support for AWS S3 as a chart repository.

    Project mention: Installing multiple helm charts in one go [Approach 2 - using helmfile] | dev.to | 2023-12-26

    $ helm plugin list NAME VERSION DESCRIPTION diff 3.8.1 Preview helm upgrade changes as a diff helm-git 0.12.0 Get non-packaged Charts directly from Git. s3 0.14.0 Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3 secrets 4.1.1 This plugin provides secrets values encryption for Helm charts secure storing

  • ftpserver

    Golang based autonomous FTP server with SFTP, S3, Dropbox, and Google Drive connectors.

  • storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

  • mort

    Storage and image processing server written in Go

  • Walrus

    🔥 Fast, Secure and Reliable System Backup, Set up in Minutes. (by Clivern)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-27.

Go S3 related posts

Index

What are some of the best open-source S3 projects in Go? This list will help you:

Project Stars
1 minio 43,252
2 rclone 42,895
3 seaweedfs 20,598
4 thanos 12,479
5 litestream 9,625
6 juicefs 9,518
7 SFTPGo 7,778
8 goofys 4,975
9 wal-g 2,933
10 mc 2,640
11 S3Scanner 2,334
12 s5cmd 2,237
13 minio-go 2,200
14 docker-volume-backup 1,403
15 rill 1,221
16 clickhouse-backup 1,111
17 s3-benchmark 755
18 csi-s3 720
19 helm-s3 534
20 ftpserver 533
21 storage 505
22 mort 498
23 Walrus 477
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com