SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Go S3 Projects
-
-
rclone
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
-
thanos
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
-
-
SFTPGo
Full-featured and highly configurable SFTP, HTTP/S, FTP/S and WebDAV server - S3, Google Cloud Storage, Azure Blob
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
-
-
-
docker-volume-backup
Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox or SSH compatible storage
-
rill
Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code. (by rilldata)
-
clickhouse-backup
Tool for easy ClickHouse backup and restore using object storage for backup files.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: JuiceFS 1.2 Beta 1: Gateway Upgrade, Enhanced Multi-User Permission Management | dev.to | 2024-05-05The core service of JuiceFS Gateway is to expose the POSIX file system via the S3 API. Before v1.2, we integrated the MinIO Gateway module into our code. This module was based on the Apache license. This integration made JuiceFS operations similar to MinIO's native command minio gateway NAS. For users, JuiceFS could be seen as an implementation of MinIO Gateway. MinIO's other backend implementations include NAS and Hadoop.
rclone: a command-line program to manage files on cloud storage.
Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11Whoops: WebDAV:
https://news.ycombinator.com/item?id=39417503
SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV
I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.
It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.
Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).
I have not, but I keep meaning to collate everything I've learned into a set of useful defaults just to remind myself what settings I should be enabling and why.
Regarding Litestream, I learned pretty much all I know from their documentation: https://litestream.io/
In the past, upgrading JuiceFS versions or modifying mount parameters required users to remount the file system. This causes temporary interruptions in service. Once the underlying file system service is interrupted, it affects the normal operation of other components in the system. To avoid such issues, we’ve introduced the smooth upgrade feature in the latest release of JuiceFS 1.2. This allows users to restart or upgrade without service downtime. In this post, we’ll share how we implemented the smooth upgrade feature and how to use it.
Project mention: What you guys are hosting instead of Nextcloud? I'm sick of it. | /r/selfhosted | 2023-11-29EDIT: Thanks for the recommendations from all of you!! I've chose to use the below: - Files: sftpgo - Calendar: baikal - Notes: memos (But beware, it sends opt-out telemetry) - Network folder: webdav on sftpgo
The author needs to ask themselves: in this cloud technology stack, is there POSIX involved somewhere lower down, where I can't access it? The answer is, of course, "yes". The sort of cloud storage systems described all run on top of POSIX APIs. They provide convenience (cost efficiency is more debatable) compared to the POSIX alternative, but that's because they exist at an entirely different conceptual layer (hence the presence of POSIX anyway, just buried).
Your point about surfacing a POSIX that's actually there but hidden and thus visible to low-level Amazon employees building the S3 service which makes it invisible to S3 end customers is true but isn't the the point of the article. The author is saying there are motivations for a POSIX-like api visible also the end user.
So your explanation of stack looks like 2 layers: POSIX api <-- AWS S3 built on top of that
Author's essay is actually talking about 3 layers: POSIX <-- AWS S3 <-- POSIX
That's why the blog post has the following links to POSIX-on-top-of-S3-objects :
https://github.com/s3fs-fuse/s3fs-fuse
https://github.com/kahing/goofys
https://www.cuno.io/
Project mention: WAL-G 3.0.0 – fast disaster recovery for Postgres | news.ycombinator.com | 2024-03-17
Project mention: GitHub issues from top Open Source Golang Repositories that you should contribute to | dev.to | 2024-01-15s5cmd - Extended character support for s3 compatible backend
I am interested in coming up with a backup plan before I get too invested in this setup. I found the docker-volume-backup project that looks like it might be a possible solution. However I'm not sure how to implement it using docker swarm since I am new to all of this. I would be interested in learning what backup solution you use for your docker swarm servers.
Project mention: S3 Benchmark: Measure Amazon S3's performance from any location | news.ycombinator.com | 2024-03-23
Project mention: Installing multiple helm charts in one go [Approach 2 - using helmfile] | dev.to | 2023-12-26$ helm plugin list NAME VERSION DESCRIPTION diff 3.8.1 Preview helm upgrade changes as a diff helm-git 0.12.0 Get non-packaged Charts directly from Git. s3 0.14.0 Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3 secrets 4.1.1 This plugin provides secrets values encryption for Helm charts secure storing
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Go S3 related posts
-
JuiceFS 1.2 Beta 1: Gateway Upgrade, Enhanced Multi-User Permission Management
-
How (and why) to run SQLite in production
-
S3 Benchmark: Measure Amazon S3's performance from any location
-
SQLite3 Replication: A Wizard's Guide🧙🏽
-
Show HN: Stree – tree view of S3 bucket
-
A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes
-
Ask HN: Time travel" into a SQLite database using the WAL files?
-
A note from our sponsor - SaaSHub
www.saashub.com | 10 May 2024
Index
What are some of the best open-source S3 projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | minio | 44,434 |
| 2 | rclone | 43,959 |
| 3 | seaweedfs | 21,123 |
| 4 | thanos | 12,599 |
| 5 | litestream | 10,026 |
| 6 | juicefs | 9,836 |
| 7 | SFTPGo | 8,185 |
| 8 | goofys | 5,042 |
| 9 | wal-g | 3,053 |
| 10 | mc | 2,698 |
| 11 | S3Scanner | 2,398 |
| 12 | s5cmd | 2,349 |
| 13 | minio-go | 2,285 |
| 14 | docker-volume-backup | 1,520 |
| 15 | rill | 1,359 |
| 16 | clickhouse-backup | 1,151 |
| 17 | s3-benchmark | 776 |
| 18 | csi-s3 | 741 |
| 19 | ftpserver | 562 |
| 20 | helm-s3 | 542 |
| 21 | storage | 521 |
| 22 | mort | 500 |
| 23 | Walrus | 477 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com