The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 Go Storage Projects
-
-
thanos
Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
storj
Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
lotusdb
Most advanced key-value database written in Go, extremely fast, compatible with LSM tree and B+ tree.
-
-
-
IceFireDB
@IceFireLabs -> IceFireDB is a database built for web3.0 It strives to fill the gap between web2 and web3.0 with a friendly database experience, making web3 application data storage more convenient, and making it easier for web2 applications to achieve decentralization and data immutability.
-
-
-
-
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes | news.ycombinator.com | 2024-02-22Looks like minio added this in 2022:
https://github.com/minio/minio/pull/15433
Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).
I have some experience with Ceph, both for work, and with homelab-y stuff.
First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes.
For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines.
Also, Ceph does prefer physical access to disks (similar to ZFS).
And you do need decent networking connectivity - I think that's the main thing people think of, when they think of high hardware requirements for Ceph. Ideally 10Gbe at the minimum - although more if you want higher performance - there can be a lot of network traffic, particularly with things like backfill. (25Gbps if you can find that gear cheap for homelab - 50Gbps is a technological dead-end. 100Gbps works well).
But honestly, for a homelab, a cheap mini PC or NUC with 10Gbe will work fine, and you should get acceptable performance, and it'll be good for learning.
You can install Ceph directly on bare-metal, or if you want to do the homelab k8s route, you can use Rook (https://rook.io/).
Hope this helps, and good luck! Let me know if you have any other questions.
Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18Support for Kerberos keytab files
Super intriguing. Thanks for sharing!
It reminds me a bit of an early Go project called Upspin [1]. And also a bit of Solid [2]. Did you get any inspiration from them?
What excites me about your project is that you're addressing the elephant in the room when it comes to data sovereignty (~nobody wants to self-host a personal database but their personal devices aren't publicly accessible) in an elegant way.
By storing the data on my personal device and (presumably?) paying for a managed relay (and maybe an encrypted backup), I can keep my data in my physical possession, but I won't have to host anything on my own. Is that the idea?
https://upspin.io/
Project mention: Do you donate your CPU time, storage, or bandwidth from your homelab to any altruistic purpose? | /r/homelab | 2023-10-16I did storj.io but was not profitable and the support was worthless. Did join NTP Pool (as I have a stratum 2 GPS NTP) but the power supply died and I haven't been able to get time to fix it.
Project mention: GitHub issues from top Open Source Golang Repositories that you should contribute to | dev.to | 2024-01-15s5cmd - Extended character support for s3 compatible backend
Project mention: LotusDB: Go's most advanced key-value store, fast, advantages of LSM and B+ tree | news.ycombinator.com | 2023-10-10
Project mention: Show HN: Winglang – a new Cloud-Oriented programming language | news.ycombinator.com | 2023-12-06> not really.. as soon as you need cloud resources that are not in your k8s cluster, you end up with the cloud in your development loop.
https://github.com/localstack/awscli-local
https://github.com/fsouza/fake-gcs-server
https://cloud.google.com/sdk/docs/downloads-docker
I’ve used all of these locally to great success.
Glad to hear you guys want to interop with npm. Is wing going to be a superset of js then?
Project mention: Openshift pipeline to copy postgres DB from pod running in production project to a pod running in test project. How can this be achieved? | /r/openshift | 2023-05-29
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Go Storage related posts
- Distribute Artifacts Across OCI Registries
- A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes
- Ceph: A Journey to 1 TiB/s
- Filecoin Foundation Successfully Deploys IPFS in Space
- Running stateful workloads on Kubernetes with Rook Ceph
- OCI image from dockerfile
- Simulate multi-nodes configuration
-
A note from our sponsor - WorkOS
workos.com | 26 Apr 2024
Index
What are some of the best open-source Storage projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | minio | 44,220 |
| 2 | thanos | 12,577 |
| 3 | rook | 11,905 |
| 4 | juicefs | 9,791 |
| 5 | drive | 6,628 |
| 6 | upspin | 6,225 |
| 7 | storj | 2,869 |
| 8 | mc | 2,690 |
| 9 | s5cmd | 2,324 |
| 10 | lotusdb | 1,944 |
| 11 | oras | 1,256 |
| 12 | operator | 1,092 |
| 13 | IceFireDB | 1,075 |
| 14 | ObjectBox Go Database | 1,032 |
| 15 | fake-gcs-server | 959 |
| 16 | csi-digitalocean | 563 |
| 17 | storage | 520 |
| 18 | directpv | 517 |
| 19 | hwameistor | 510 |
| 20 | mort | 499 |
| 21 | volsync | 456 |
| 22 | ArtiVC | 281 |
| 23 | vasedb | 252 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com