Go Storage

Open-source Go projects categorized as Storage

Top 23 Go Storage Projects

  • minio

    The Object Store for AI Data Infrastructure

    Project mention: A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes | news.ycombinator.com | 2024-02-22

    Looks like minio added this in 2022:

    https://github.com/minio/minio/pull/15433

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    Project mention: Looking for a way to remote in to K's of raspberry pi's... | /r/sysadmin | 2023-12-10

    Monitoring = netdata on each RPi https://www.netdata.cloud/ binded to the vpn interface being scraped into a prometeus thaons https://thanos.io/ setup with grafana to give management the Green all is good screens (very important).

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • rook

    Storage Orchestration for Kubernetes

    Project mention: Ceph: A Journey to 1 TiB/s | news.ycombinator.com | 2024-01-19

    I have some experience with Ceph, both for work, and with homelab-y stuff.

    First, bear in mind that Ceph is a distributed storage system - so the idea is that you will have multiple nodes.

    For learning, you can definitely virtualise it all on a single box - but you'll have a better time with discrete physical machines.

    Also, Ceph does prefer physical access to disks (similar to ZFS).

    And you do need decent networking connectivity - I think that's the main thing people think of, when they think of high hardware requirements for Ceph. Ideally 10Gbe at the minimum - although more if you want higher performance - there can be a lot of network traffic, particularly with things like backfill. (25Gbps if you can find that gear cheap for homelab - 50Gbps is a technological dead-end. 100Gbps works well).

    But honestly, for a homelab, a cheap mini PC or NUC with 10Gbe will work fine, and you should get acceptable performance, and it'll be good for learning.

    You can install Ceph directly on bare-metal, or if you want to do the homelab k8s route, you can use Rook (https://rook.io/).

    Hope this helps, and good luck! Let me know if you have any other questions.

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: South Korea's No.1 Search Engine Chose JuiceFS over Alluxio for AI Storage | dev.to | 2024-01-18

    Support for Kerberos keytab files

  • drive

    Google Drive client for the commandline

  • upspin

    Upspin: A framework for naming everyone's everything.

    Project mention: I Moved My Blog from IPFS to a Server | news.ycombinator.com | 2024-01-31

    Super intriguing. Thanks for sharing!

    It reminds me a bit of an early Go project called Upspin [1]. And also a bit of Solid [2]. Did you get any inspiration from them?

    What excites me about your project is that you're addressing the elephant in the room when it comes to data sovereignty (~nobody wants to self-host a personal database but their personal devices aren't publicly accessible) in an elegant way.

    By storing the data on my personal device and (presumably?) paying for a managed relay (and maybe an encrypted backup), I can keep my data in my physical possession, but I won't have to host anything on my own. Is that the idea?

    https://upspin.io/

  • storj

    Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

    Project mention: Do you donate your CPU time, storage, or bandwidth from your homelab to any altruistic purpose? | /r/homelab | 2023-10-16

    I did storj.io but was not profitable and the support was worthless. Did join NTP Pool (as I have a stratum 2 GPS NTP) but the power supply died and I haven't been able to get time to fix it.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • mc

    Simple | Fast tool to manage MinIO clusters :cloud:

  • s5cmd

    Parallel S3 and local filesystem execution tool.

    Project mention: GitHub issues from top Open Source Golang Repositories that you should contribute to | dev.to | 2024-01-15

    s5cmd - Extended character support for s3 compatible backend

  • lotusdb

    Most advanced key-value store written in Go, extremely fast, compatible with LSM tree and B+ tree, optimization of badger and bbolt.

    Project mention: LotusDB: Go's most advanced key-value store, fast, advantages of LSM and B+ tree | news.ycombinator.com | 2023-10-10
  • oras

    OCI registry client - managing content like artifacts, images, packages

    Project mention: Distribute Artifacts Across OCI Registries | news.ycombinator.com | 2024-03-13
  • operator

    Simple Kubernetes Operator for MinIO clusters :computer:

  • IceFireDB

    @IceFireLabs -> IceFireDB is a database built for web3.0 It strives to fill the gap between web2 and web3.0 with a friendly database experience, making web3 application data storage more convenient, and making it easier for web2 applications to achieve decentralization and data immutability.

  • ObjectBox Go Database

    Embedded Go Database, the fast alternative to SQLite, gorm, etc.

  • fake-gcs-server

    Google Cloud Storage emulator & testing library.

    Project mention: Show HN: Winglang – a new Cloud-Oriented programming language | news.ycombinator.com | 2023-12-06

    > not really.. as soon as you need cloud resources that are not in your k8s cluster, you end up with the cloud in your development loop.

    https://github.com/localstack/awscli-local

    https://github.com/fsouza/fake-gcs-server

    https://cloud.google.com/sdk/docs/downloads-docker

    I’ve used all of these locally to great success.

    Glad to hear you guys want to interop with npm. Is wing going to be a superset of js then?

  • csi-digitalocean

    A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage

  • storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

  • directpv

    Simple Kubernetes CSI driver for Direct Attached Storage :minidisc:

  • hwameistor

    Hwameistor is an HA local storage system for cloud-native stateful workloads.

  • mort

    Storage and image processing server written in Go

  • volsync

    Asynchronous data replication for Kubernetes volumes

    Project mention: Openshift pipeline to copy postgres DB from pod running in production project to a pod running in test project. How can this be achieved? | /r/openshift | 2023-05-29
  • ArtiVC

    A version control system to manage large files.

  • vasedb

    VaseDB is a NoSQL that supports multiple data types and transactions.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-13.

Go Storage related posts

Index

What are some of the best open-source Storage projects in Go? This list will help you:

Project Stars
1 minio 43,980
2 thanos 12,561
3 rook 11,864
4 juicefs 9,752
5 drive 6,625
6 upspin 6,218
7 storj 2,859
8 mc 2,683
9 s5cmd 2,299
10 lotusdb 1,939
11 oras 1,238
12 operator 1,081
13 IceFireDB 1,072
14 ObjectBox Go Database 1,025
15 fake-gcs-server 950
16 csi-digitalocean 563
17 storage 517
18 directpv 515
19 hwameistor 505
20 mort 499
21 volsync 446
22 ArtiVC 281
23 vasedb 252
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com