The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 aws-s3 Open-Source Projects
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
formidable
The most used, flexible, fast and streaming parser for multipart form data. Supports uploading to serverless environments, AWS S3, Azure, GCP or the filesystem. Used in production.
-
-
SaaS Boilerplate
Build your own SaaS business with SaaS boilerplate. Productive stack: React, Material-UI, Next, MobX, WebSockets, Express, Node, Mongoose, MongoDB. Written with TypeScript.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
S3 Server
Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.
-
termscp
🖥 A feature rich terminal UI file transfer and explorer with support for SCP/SFTP/FTP/S3/SMB
-
Udacity-Data-Engineering-Projects
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
-
Meteor-Files
🚀 Upload files via DDP or HTTP to ☄️ Meteor server FS, AWS, GridFS, DropBox or Google Drive. Fast, secure and robust.
-
-
-
S3Mock
A simple mock implementation of the AWS S3 API startable as Docker image, TestContainer, JUnit 4 rule, JUnit Jupiter extension or TestNG listener
-
-
tus-node-server
Node.js tus server, standalone or integrable in any framework, with disk, S3, and GGC stores.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Go does not natively support the use of migrations, but we could use the ORM that has this functionality, such as GORM which is the most used by the community, but We can use migrations without using an ORM, for this we will use the golang-migrate package.
One note of caution, though: Amplify uses a frontend-only Cognito integration that stores long-lived, never-rotating refresh tokens in browser storage, where any XSS vulnerability would have access to them. A more secure approach is to implement a couple of backend API routes to store the refresh tokens in `HttpOnly` cookies instead, which I outlined here (option 1 in your case to support SSO). I'll probably open source a solution to do this early next year so we don't all have to keep reinventing this wheel (probably why AWS calls their conference re:invent).
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.
So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
https://github.com/kahing/goofys
https://www.cuno.io/
Instantiate a multipart/form-data parser using a library called formidable
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.
So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
https://github.com/kahing/goofys
https://www.cuno.io/
# Download the LakeFS binary wget https://github.com/treeverse/lakeFS/releases/latest/download/lakefs # Make the binary executable chmod +x lakefs # Initialize LakeFS with S3 as the storage backend ./lakefs init --backend s3 --s3-gateway-endpoint --s3-region --s3-force-path-style --s3-access-key --s3-secret-key
Project mention: WAL-G 3.0.0 – fast disaster recovery for Postgres | news.ycombinator.com | 2024-03-17
Here is something on GitHub that roughly follows this paradigm, and was made to use SES as the "email server".
Project mention: Show HN: OpenSign – The open source alternative to DocuSign | news.ycombinator.com | 2023-10-28> Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.
Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing
I've actually been looking for more open alternatives, but haven't found much.
Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)
There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/
The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs
Project mention: Ask HN: AWS S3, Cloudflare R2, GCS, Wasabi, or B2? | news.ycombinator.com | 2024-04-01
Project mention: Termscp TUI terminal file transfer and explorer with support for SCP/SFTP/FTP/S3 | /r/commandline | 2023-04-30
Supabase contributed some advanced features from the Node implementation of TUS Spec including distributed locks, max file size, expiration extension and numerous bug fixes:
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
aws-s3 related posts
- Ask HN: AWS S3, Cloudflare R2, GCS, Wasabi, or B2?
- Show HN: Versatile email infrastructure on AWS serverless
- Email Inbox on AWS Serverless
- Is Posix Outdated?
- cls3, Búsqueda y Eliminación Masiva de Buckets S3
- R2 slow PUT file transfer
- Cloud Backed SQLite
-
A note from our sponsor - WorkOS
workos.com | 24 Apr 2024
Index
What are some of the best open-source aws-s3 projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | migrate | 13,946 |
| 2 | amplify-js | 9,363 |
| 3 | s3fs-fuse | 8,065 |
| 4 | formidable | 6,943 |
| 5 | goofys | 5,031 |
| 6 | lakeFS | 4,058 |
| 7 | SaaS Boilerplate | 3,899 |
| 8 | wal-g | 3,038 |
| 9 | 0x4447_product_s3_email | 3,010 |
| 10 | mc | 2,690 |
| 11 | S3 Server | 1,627 |
| 12 | s3proxy | 1,595 |
| 13 | termscp | 1,337 |
| 14 | Udacity-Data-Engineering-Projects | 1,295 |
| 15 | Meteor-Files | 1,114 |
| 16 | minio-java | 1,028 |
| 17 | s3-sync-action | 1,004 |
| 18 | minio-js | 880 |
| 19 | Rome | 815 |
| 20 | S3Mock | 765 |
| 21 | minio-py | 763 |
| 22 | tus-node-server | 723 |
| 23 | Go-Clean-Architecture-REST-API | 641 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com