S3

Open-source projects categorized as S3
Edit details
Language: + Go + Python + C++ + Ruby + Rust + TypeScript + JavaScript + PHP
Topics: AWS aws-s3 Storage Golang Python
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
Sponsored

Top 23 S3 Open-Source Projects

  • minio

    The Object Store for AI Data Infrastructure

    • Project mention: A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes | news.ycombinator.com | 2024-02-22

    Looks like minio added this in 2022:

    https://github.com/minio/minio/pull/15433

  • rclone

    "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files

    • Project mention: Supabase Storage: now supports the S3 protocol | dev.to | 2024-04-18

    rclone: a command-line program to manage files on cloud storage.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    • Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11

    Whoops: WebDAV:

    https://news.ycombinator.com/item?id=39417503

    SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV

    I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.

    It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.

  • siyuan

    A privacy-first, self-hosted, fully open source personal knowledge management software, written in typescript and golang.

    • Project mention: A structured note-taking app for personal use | news.ycombinator.com | 2023-12-21

    Try SiYuan Note. It's free and open source local-first mix of Notion and Obsidian.

    https://github.com/siyuan-note/siyuan

  • airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    • Project mention: Launch HN: Bracket (YC W22) – Two-Way Sync Between Salesforce and Postgres | news.ycombinator.com | 2023-12-12

    I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us.

    It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.

  • Ceph

    Ceph is a distributed object, block, and file storage platform

    • Project mention: First time user sturggles | /r/ceph | 2023-06-24

    curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadmchmod a+x cephadm./cephadm bootstrap --mon-ip 192.168.1.41

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    • Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10

    Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • awesome-aws

    A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

  • litestream

    Streaming replication for SQLite.

    • Project mention: Ask HN: SQLite in Production? | news.ycombinator.com | 2024-04-07

    I have not, but I keep meaning to collate everything I've learned into a set of useful defaults just to remind myself what settings I should be enabling and why.

    Regarding Litestream, I learned pretty much all I know from their documentation: https://litestream.io/

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    • Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

    Support for Kerberos keytab files

  • Filestash

    🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...

    • Project mention: Ask HN: What Underrated Open Source Project Deserves More Recognition? | news.ycombinator.com | 2024-03-07

  • SFTPGo

    Fully featured and highly configurable SFTP server with optional HTTP/S, FTP/S and WebDAV support - S3, Google Cloud Storage, Azure Blob

    • Project mention: What you guys are hosting instead of Nextcloud? I'm sick of it. | /r/selfhosted | 2023-11-29

    EDIT: Thanks for the recommendations from all of you!! I've chose to use the below: - Files: sftpgo - Calendar: baikal - Notes: memos (But beware, it sends opt-out telemetry) - Network folder: webdav on sftpgo

  • s3fs-fuse

    FUSE-based file system backed by Amazon S3

    • Project mention: Is Posix Outdated? | news.ycombinator.com | 2023-10-19

    The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

    Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.

    So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

    Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

    That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

    https://github.com/s3fs-fuse/s3fs-fuse

    https://github.com/kahing/goofys

    https://www.cuno.io/

  • Moto

    A library that allows you to easily mock out tests based on AWS infrastructure.

    • Project mention: OpenTF Announces Fork of Terraform | news.ycombinator.com | 2023-08-25

    > OpenMoto

    I dunno if you're trying to play on "hashimoto" but https://github.com/getmoto/moto#readme would be a prime name collision for any such "OpenMoto" name

    But yes, please, to adopting Vault. I don't have a horse in the race about Consul but my suspicion is such an effort would only be worthwhile if trying to adopt Nomad, too, which I gravely doubt

  • KodExplorer

    A web based file manager,web IDE / browser based code editor

  • goofys

    a high-performance, POSIX-ish Amazon S3 file system written in Go

    • Project mention: Is Posix Outdated? | news.ycombinator.com | 2023-10-19

    The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).

    Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.

    So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that

    Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX

    That's why the blog post has the following links to POSIX-on-top-of-S3-objects :

    https://github.com/s3fs-fuse/s3fs-fuse

    https://github.com/kahing/goofys

    https://www.cuno.io/

  • Backup

    Easy full stack backup operations on UNIX-like systems.

    • Project mention: Backup VS database - a user suggested alternative | libhunt.com/r/backup | 2023-06-21

  • s3cmd

    Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).

    • Project mention: Amazon S3 Tools: Command Line S3 Client and S3 Backup | news.ycombinator.com | 2024-01-26

  • mountpoint-s3

    A simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system.

    • Project mention: Row Zero and Viewport Data Streaming | news.ycombinator.com | 2024-04-04

    ... or does "S3 file system" mean https://github.com/awslabs/mountpoint-s3 - a Rust project by AWS Labs that provides "a simple, high-throughput file client for mounting an Amazon S3 bucket as a local file system" ?

  • Replibyte

    Seed your development database with real data ⚡️

  • wal-e

    Continuous Archiving for Postgres

    • Project mention: Run PostgreSQL. The Kubernetes Way | news.ycombinator.com | 2023-09-22

    See the GitHub: https://github.com/wal-e/wal-e

    Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

    Seems to be obsolete due to a lack of interest and contributions.

  • Shrine

    File Attachment toolkit for Ruby applications

    • Project mention: [Poll] How do you handle file attachments in your Rails app in 2023? | /r/rails | 2023-07-05

    Shrine is great, has more features and a simpler API (imo) than ActiveStorage https://shrinerb.com

  • smart_open

    Utils for streaming large files (S3, HDFS, gzip, bz2...)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

S3 related posts

Index

What are some of the best open-source S3 projects? This list will help you:

Project Stars
1 minio 44,094
2 rclone 43,720
3 seaweedfs 21,013
4 siyuan 15,733
5 airbyte 13,923
6 Ceph 13,197
7 thanos 12,577
8 awesome-aws 12,147
9 litestream 9,964
10 juicefs 9,791
11 Filestash 9,414
12 SFTPGo 8,117
13 s3fs-fuse 8,065
14 Moto 7,374
15 KodExplorer 6,167
16 goofys 5,031
17 Backup 4,817
18 s3cmd 4,418
19 mountpoint-s3 4,003
20 Replibyte 3,990
21 wal-e 3,423
22 Shrine 3,142
23 smart_open 3,091

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com