Open-source projects categorized as S3 | Edit details
Related topics: #AWS #Golang #Go #Storage #aws-s3

Top 23 S3 Open-Source Projects

  • GitHub repo minio

    High Performance, Kubernetes Native Object Storage

    Project mention: MinIO Server on Pi with Autorestic as Backup Solution | reddit.com/r/selfhosted | 2021-06-16

    Yes but only ARM64. Refer to https://github.com/minio/minio#gnulinux for download link.

  • GitHub repo rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    Project mention: How do you handle the backing-up of all data when self-hosting? | reddit.com/r/selfhosted | 2021-06-14

    rclone.org into a cheap but reliable cloud solution (in my case backblaze.com) as someone else already said, there is no 'set it and forget it' backup, but a decently monitored script with some alert only when things go wrong comes close. You can fully encrypt client side, the cloud provider doesn't even see as much as filenames - and it is easy to restore files back to any system you can get an rclone client for.

  • GitHub repo Seaweed File System

    SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

    Project mention: Three node system needs low latency distributed storage | reddit.com/r/homelab | 2021-06-17

    SeaweedFS has been bookmarked for a while but I haven't had a chance to play with it. But 100ms between nodes is an architecture problem, not a software problem (IMO).

  • GitHub repo awesome-aws

    A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

    Project mention: There are 40,000+ quality AWS open source repositories on GitHub but are completely unorganized. I made a search engine and browser for all of them, all curated carefully with 1000+ filters. | reddit.com/r/sysadmin | 2021-06-06

    There is also https://github.com/donnemartin/awesome-aws

  • GitHub repo thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    Project mention: Searching for advanced resources about mmap-ing in Go! Any books/papers/docs? | reddit.com/r/golang | 2021-06-13

    Memory mapping is an effective way to bind portion file into memory and using it as it's your own area in the memory. I happen to be a heavy Go user of memory mapping for some time series databases we maintain (Prometheus, Thanos). We even created a useful helper for it (stable, battle-tested, feel free to use https://github.com/prometheus/prometheus/blob/main/tsdb/fileutil/mmap.go#L22).

  • GitHub repo s3fs-fuse

    FUSE-based file system backed by Amazon S3

    Project mention: Moving my home media library from iTunes to Jellyfin and Infuse | news.ycombinator.com | 2021-06-10

    > Are there any approaches to throw your library behind authed CDN or aws s3 with a frontend ios/android/desktop app to get rid of those fancy subscription models?!

    You can use an S3 compatible storage provider and either mount it via NFS[1] or s3fs[2], and point Jellyfin to it.

    [1] https://docs.aws.amazon.com/storagegateway/latest/userguide/...

    [2] https://github.com/s3fs-fuse/s3fs-fuse

  • GitHub repo Moto

    A library that allows you to easily mock out tests based on AWS infrastructure.

    Project mention: Ask HN: Does anyone else find the AWS Lambda developer experience poor? | news.ycombinator.com | 2021-04-18

    Not a full solution, but when I was doing this I really got to love the awslogs utility:


    It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)

    Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.

    Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.


  • GitHub repo Backup

    Easy full stack backup operations on UNIX-like systems.

    Project mention: Backup Postgresql Docker | reddit.com/r/devops | 2021-04-16

    I used to be a huge fan of https://github.com/backup/backup, but unfortunately it doesn't get updates. Still works though. I ran hundreds of gigs each night through that tool. There's a golang version re-write that is limited on the features - https://github.com/huacnlee/gobackup.

  • GitHub repo Filestash

    🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...

    Project mention: Web-based collaborative file manager with external auth and storage backend | reddit.com/r/selfhosted | 2021-05-13

    That said there's something that I found that looks similar to what I'm looking for, and it's called [filestash](https://www.filestash.app/), but it doesn't seem to support OneDrive as a backend nor external auth. Nextcloud also seems to support some external backends, but not OneDrive aswell, but it does seem to support external auth.

  • GitHub repo s3cmd

    Official s3cmd repo -- Command line tool for managing Amazon S3 and CloudFront services

    Project mention: Syncing media between two Plex servers | reddit.com/r/PleX | 2021-03-01
  • GitHub repo goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    Project mention: AWS Developer Forums: S3 Block Devices | news.ycombinator.com | 2020-12-30

    Yes, still a bad idea, but if you really need to, then goofys is probably the best way to do it


  • GitHub repo juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: "JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environment." | reddit.com/r/programming | 2021-04-20
  • GitHub repo wal-e

    Continuous Archiving for Postgres

    Project mention: Automate pg_basebackup in linux | reddit.com/r/PostgreSQL | 2021-06-05

    Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E

  • GitHub repo Shrine

    File Attachment toolkit for Ruby applications

    Project mention: Image Uploading with Shrine | dev.to | 2021-04-23

    Once I knew I wanted to upload images I stated to look around for different ways to do so. I came across a gem named shrine which can be found here https://shrinerb.com/. This is were the hard part came in. Aside from installing the gem like normal, there was a second step that needed to be done to allow images to be rendered. I needed to install https://imagemagick.org/index.php ImageMagick onto my system. After this I was able to add images. ​

  • GitHub repo SFTPGo

    Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    Project mention: SFTPGo 2.1.0 released with many new features and improvements | reddit.com/r/linux | 2021-06-16
  • GitHub repo smart_open

    Utils for streaming large files (S3, HDFS, gzip, bz2...)

    Project mention: Fsspec: Filesystem Interfaces for Python | news.ycombinator.com | 2021-06-15

    See also smart_open: https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579

  • GitHub repo mc

    MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.

  • GitHub repo wal-g

    Archival and Restoration for Postgres

    Project mention: Automate pg_basebackup in linux | reddit.com/r/PostgreSQL | 2021-06-05

    Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E

  • GitHub repo S3-Uploads

    The WordPress Plugin to Store Uploads on Amazon S3

    Project mention: Part 4 - Wordpress EC2 instance in ASG with RDS database and ALB- Awesome AWS CDK | dev.to | 2021-05-24

    Create an S3 bucket using the CDK and use this Wordpress plugin to sync Wordpress uploads to AWS S3. Hint: You install this plugin via the user script.

  • GitHub repo LeoFS

    The LeoFS Storage System

    Project mention: Ask HN: How would you store 10PB of data for your startup today? | news.ycombinator.com | 2021-04-23

    I think if I _had_ to decide (I'm not the best informed person on the matter) I'd lean towards leofs[1].

    I only read about it, but never used it.

    It advertises itself as exabyte scalable and provides s3 and nfs access.

    [1] https://leo-project.net/leofs/

  • GitHub repo Foundatio

    Pluggable foundation blocks for building distributed apps.

  • GitHub repo minio-go

    MinIO Client SDK for Go

  • GitHub repo cyberduck

    Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, Backblaze B2, Microsoft Azure & OneDrive and OpenStack Swift file transfer client for Mac and Windows.

    Project mention: Does installing cloud drive potentially. | reddit.com/r/privacytoolsIO | 2021-06-15

    If you’re installing the app, as with any other app you’d have to trust it. You might want to use webdav to connect to it in your file manager rather than using the app or use cyberduck, afaik it also has cyptomator built in or a encryption based on iryptomator. On Linux you could use it’s cli version

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-06-17.


What are some of the best open-source S3 projects? This list will help you:

Project Stars
1 minio 28,121
2 rclone 27,605
3 Seaweed File System 12,177
4 awesome-aws 9,377
5 thanos 8,572
6 s3fs-fuse 5,281
7 Moto 4,707
8 Backup 4,661
9 Filestash 4,653
10 s3cmd 3,637
11 goofys 3,624
12 juicefs 3,308
13 wal-e 3,250
14 Shrine 2,829
15 SFTPGo 2,613
16 smart_open 2,079
17 mc 1,818
18 wal-g 1,786
19 S3-Uploads 1,486
20 LeoFS 1,412
21 Foundatio 1,400
22 minio-go 1,311
23 cyberduck 1,231
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives