Top 23 S3 Open-Source Projects
High Performance, Kubernetes Native Object StorageProject mention: MinIO Server on Pi with Autorestic as Backup Solution | reddit.com/r/selfhosted | 2021-06-16
Yes but only ARM64. Refer to https://github.com/minio/minio#gnulinux for download link.
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex FilesProject mention: How do you handle the backing-up of all data when self-hosting? | reddit.com/r/selfhosted | 2021-06-14
rclone.org into a cheap but reliable cloud solution (in my case backblaze.com) as someone else already said, there is no 'set it and forget it' backup, but a decently monitored script with some alert only when things go wrong comes close. You can fully encrypt client side, the cloud provider doesn't even see as much as filenames - and it is easy to restore files back to any system you can get an rclone client for.
Scout APM - Leading-edge performance monitoring starting at $39/month. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.Project mention: Three node system needs low latency distributed storage | reddit.com/r/homelab | 2021-06-17
SeaweedFS has been bookmarked for a while but I haven't had a chance to play with it. But 100ms between nodes is an architecture problem, not a software problem (IMO).
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.Project mention: There are 40,000+ quality AWS open source repositories on GitHub but are completely unorganized. I made a search engine and browser for all of them, all curated carefully with 1000+ filters. | reddit.com/r/sysadmin | 2021-06-06
There is also https://github.com/donnemartin/awesome-aws
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.Project mention: Searching for advanced resources about mmap-ing in Go! Any books/papers/docs? | reddit.com/r/golang | 2021-06-13
Memory mapping is an effective way to bind portion file into memory and using it as it's your own area in the memory. I happen to be a heavy Go user of memory mapping for some time series databases we maintain (Prometheus, Thanos). We even created a useful helper for it (stable, battle-tested, feel free to use https://github.com/prometheus/prometheus/blob/main/tsdb/fileutil/mmap.go#L22).
FUSE-based file system backed by Amazon S3Project mention: Moving my home media library from iTunes to Jellyfin and Infuse | news.ycombinator.com | 2021-06-10
> Are there any approaches to throw your library behind authed CDN or aws s3 with a frontend ios/android/desktop app to get rid of those fancy subscription models?!
You can use an S3 compatible storage provider and either mount it via NFS or s3fs, and point Jellyfin to it.
A library that allows you to easily mock out tests based on AWS infrastructure.Project mention: Ask HN: Does anyone else find the AWS Lambda developer experience poor? | news.ycombinator.com | 2021-04-18
Not a full solution, but when I was doing this I really got to love the awslogs utility:
It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)
Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.
Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.
Run Linux Software Faster and Safer than Linux with Unikernels.
Easy full stack backup operations on UNIX-like systems.Project mention: Backup Postgresql Docker | reddit.com/r/devops | 2021-04-16
I used to be a huge fan of https://github.com/backup/backup, but unfortunately it doesn't get updates. Still works though. I ran hundreds of gigs each night through that tool. There's a golang version re-write that is limited on the features - https://github.com/huacnlee/gobackup.
🦄 A modern web client for SFTP, S3, FTP, WebDAV, Git, Minio, LDAP, CalDAV, CardDAV, Mysql, Backblaze, ...Project mention: Web-based collaborative file manager with external auth and storage backend | reddit.com/r/selfhosted | 2021-05-13
That said there's something that I found that looks similar to what I'm looking for, and it's called [filestash](https://www.filestash.app/), but it doesn't seem to support OneDrive as a backend nor external auth. Nextcloud also seems to support some external backends, but not OneDrive aswell, but it does seem to support external auth.
Official s3cmd repo -- Command line tool for managing Amazon S3 and CloudFront servicesProject mention: Syncing media between two Plex servers | reddit.com/r/PleX | 2021-03-01
a high-performance, POSIX-ish Amazon S3 file system written in Go
JuiceFS is a distributed POSIX file system built on top of Redis and S3.Project mention: "JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environment." | reddit.com/r/programming | 2021-04-20
Continuous Archiving for PostgresProject mention: Automate pg_basebackup in linux | reddit.com/r/PostgreSQL | 2021-06-05
Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E
File Attachment toolkit for Ruby applicationsProject mention: Image Uploading with Shrine | dev.to | 2021-04-23
Once I knew I wanted to upload images I stated to look around for different ways to do so. I came across a gem named shrine which can be found here https://shrinerb.com/. This is were the hard part came in. Aside from installing the gem like normal, there was a second step that needed to be done to allow images to be rendered. I needed to install https://imagemagick.org/index.php ImageMagick onto my system. After this I was able to add images.
Fully featured and highly configurable SFTP server with optional FTP/S and WebDAV support - S3, Google Cloud Storage, Azure BlobProject mention: SFTPGo 2.1.0 released with many new features and improvements | reddit.com/r/linux | 2021-06-16
Utils for streaming large files (S3, HDFS, gzip, bz2...)Project mention: Fsspec: Filesystem Interfaces for Python | news.ycombinator.com | 2021-06-15
See also smart_open: https://github.com/RaRe-Technologies/smart_open which might be more user-friendly? Never used it myself but it was on HN before. Discussion on their bugtracker: https://github.com/RaRe-Technologies/smart_open/issues/579
MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.
Archival and Restoration for PostgresProject mention: Automate pg_basebackup in linux | reddit.com/r/PostgreSQL | 2021-06-05
Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E
The WordPress Plugin to Store Uploads on Amazon S3Project mention: Part 4 - Wordpress EC2 instance in ASG with RDS database and ALB- Awesome AWS CDK | dev.to | 2021-05-24
Create an S3 bucket using the CDK and use this Wordpress plugin to sync Wordpress uploads to AWS S3. Hint: You install this plugin via the user script.
The LeoFS Storage SystemProject mention: Ask HN: How would you store 10PB of data for your startup today? | news.ycombinator.com | 2021-04-23
I think if I _had_ to decide (I'm not the best informed person on the matter) I'd lean towards leofs.
I only read about it, but never used it.
It advertises itself as exabyte scalable and provides s3 and nfs access.
Pluggable foundation blocks for building distributed apps.
MinIO Client SDK for Go
Cyberduck is a libre FTP, SFTP, WebDAV, Amazon S3, Backblaze B2, Microsoft Azure & OneDrive and OpenStack Swift file transfer client for Mac and Windows.Project mention: Does installing cloud drive potentially. | reddit.com/r/privacytoolsIO | 2021-06-15
If you’re installing the app, as with any other app you’d have to trust it. You might want to use webdav to connect to it in your file manager rather than using the app or use cyberduck, afaik it also has cyptomator built in or a encryption based on iryptomator. On Linux you could use it’s cli version
What are some of the best open-source S3 projects? This list will help you:
|3||Seaweed File System||12,177|