Go Storage

Open-source Go projects categorized as Storage

Top 23 Go Storage Projects

  • minio

    Multi-Cloud :cloud: Object Storage

    Project mention: Can anybody help explain the impact of this - Deprecation of the MinIO gateway | reddit.com/r/kubernetes | 2022-11-05

    On a related note, does anyone know what happened to the minio edge command mentioned here? They said that S3 gateway will live on as minio edge, but gateway was removed and edge is nowhere to be found.

  • thanos

    Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.

    Project mention: What do you guys use for outage monitoring / incident reporting? | reddit.com/r/devops | 2022-11-20

    If you don't have the budget or have a decent size team, go with Prometheus and Grafana. Be mindful to do the research, the storage for Prometheus needs to be well designed for scalability. We wanted at some point to implement this Thanos but we didn't have the bandwidth to do so.

  • Zigi

    Delete the most useless function ever: context switching.. Zigi monitors Jira and GitHub updates, pings you when PRs need approval and lets you take fast actions - all directly from Slack! Plus it reduces cycle time by up to 75%.

  • rook

    Storage Orchestration for Kubernetes

    Project mention: [HELP] PXE Boot without data loss | reddit.com/r/linuxadmin | 2022-12-04

    Third, it sounds like you're building a cluster. For this you'll either want a central file server. Or better, setup a distributed storage system. For example a Ceph cluster managed by Rook. This way you can fully wipe a single node and the system will be able to recover/replicate thed data.

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: An open-source distributed object storage service | news.ycombinator.com | 2022-12-04
  • drive

    Google Drive client for the commandline

    Project mention: 10 years since Google said to “hang tight” about Linux support for Google Drive | news.ycombinator.com | 2022-04-24

    I just want to collect the major options for Google Drive on Linux in a single comment, since a few options are scattered around:

    Insync works well, and it's 50% off for a couple more days: https://www.insynchq.com/ Not affiliated, but $15 is not a lot to pay, as opposed to waiting for something that probably won't happen.

    Rclone has support for Google Drive, and it's open source: https://rclone.org/

    There's a command line client that uses a push/pull workflow: https://github.com/odeke-em/drive It was written by a member of the Google Drive team.

    Gnome supports Google Drive, or at least used to, directly in Nautilus. I don't use Gnome, so I can't comment.

    There may be other options I've missed, but the point is that there is already good support in multiple forms. I'd be interested to know what support Google could provide that's not already available.

  • upspin

    Upspin: A framework for naming everyone's everything.

    Project mention: Upspin: A framework for naming everyone's everything. | reddit.com/r/CKsTechNews | 2022-11-16
  • storj

    Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

    Project mention: "Community" is not enough - we need commercial service providers. | reddit.com/r/Mastodon | 2022-12-07

    Storj is better.

  • Scout APM

    Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.

  • mc

    MinIO Client is a replacement for ls, cp, mkdir, diff and rsync commands for filesystems and object storage.

    Project mention: Migrating data between two online MinIO instances | dev.to | 2022-07-03

    We are going to use an official MinIO client called mc. I use macOS so installation is quite easy. The process supposes to be similar on different platforms.

  • s5cmd

    Parallel S3 and local filesystem execution tool.

    Project mention: Those using AWS, have you ever tried to use AWS Transfer Family to transfer files into an S3 bucket? Can I use python to make these uploads, and if so how do I set it up in aws? | reddit.com/r/aws | 2022-10-18

    Some folks say https://github.com/peak/s5cmd is faster than the two options above.

  • heketi

    RESTful based volume management framework for GlusterFS

  • lotusdb

    Fast k/v storage compatible with lsm tree and b+tree, inspired by SLM-DB in USENIX FAST ’19.

    Project mention: Open Source Databases in Go | reddit.com/r/golang | 2022-06-08

    lotusdb - Fast k/v database compatible with lsm and b+tree.

  • IceFireDB

    IceFireDB is a database built for web3 and web2. It strives to fill the gap between web2 and web3 with a friendly database experience, making web3 application data storage more convenient, and making it easier for web2 applications to achieve decentralization and data immutability.

    Project mention: IceFireDB-Redis-proxy:Redis database proxy meets P2P, an interesting attempt. | reddit.com/r/golang | 2022-07-17
  • oras

    OCI registry client - managing content like artifacts, images, packages

    Project mention: Looking for an artifact store for generic assets, rather than specially-formatted packages or containers. Thinking maybe ORAS, but wondering if there are other options. | reddit.com/r/devops | 2022-11-22

    oras isn't that unpopular, Helm is using it as an SDK for example. Here are other projects who are using it. https://github.com/oras-project/oras/network/dependents

  • fake-gcs-server

    Google Cloud Storage emulator & testing library.

    Project mention: What is for you the project who represents the best the power of Golang ? | reddit.com/r/golang | 2021-12-28

    Fake GCS Server: Also used this one in combination with docker-test. You have a fake GCS server being simulated via Docker

  • csi-digitalocean

    A Container Storage Interface (CSI) Driver for DigitalOcean Block Storage

    Project mention: How to deploy Docker images onto Kubernetes clusters? | reddit.com/r/kubernetes | 2022-11-15

    I see this: https://github.com/digitalocean/csi-digitalocean

  • mort

    Storage and image processing server written in Go

  • storage

    A vendor-neutral storage library for Golang: Write once, run on every storage service.

  • directpv

    Container Storage Interface (CSI) driver for Direct Attached Storage (DAS)

    Project mention: Doubts on minio deployment using kubernetes. | reddit.com/r/minio | 2022-03-11

    First I installed krew in order to install the minio and the directpv operators.

  • kalasa

    Kalasa is a simple and portable key-value storage engine.

    Project mention: A lightweight kv storage engine based on Golang. | dev.to | 2022-03-19

    The specific encryptor implementation code can be viewed in encrypted.go

  • space-daemon

    The Space Daemon packages together IPFS, Textile Threads/Buckets, and Textile Powergate (Filecoin*) into one easy to install Daemon to make it easy to build peer to peer and privacy focused apps.

  • hackpadfs

    An extensible file system abstraction for Go. File systems, composable interfaces, and test suites.

    Project mention: Read/Write FileSystem header interface for dependency injection | reddit.com/r/golang | 2022-05-29

    Have a look at hackpadfs if you like afero but prefer a "native" abstraction

  • estuary

    A custom IPFS/Filecoin node that makes it easy to pin IPFS content and make Filecoin deals. (by application-research)

    Project mention: Filecoin is falling. | reddit.com/r/filecoin | 2022-07-27

    My favorite group of Filecoin developers is the crew developing "Estuary" https://github.com/application-research/estuary. This project worked perfectly, and it is being improved all the time. I had one issue with connecting the backend to the frontend and they made a commit to fix it within a few hours! That's dedication!

  • uptoc

    A static file deployment tool that supports multiple platforms./ 一个支持多家云厂商的静态文件部署工具

    Project mention: XNXK ‘s Web Front-End Automated Publishing Practice | dev.to | 2022-07-01

    Without exception this is also a reusable workflow, and we use the Action provided by saltbo/uptoc to upload these static files to the object store and provide access to the CDN.

  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-12-07.

Go Storage related posts

Index

What are some of the best open-source Storage projects in Go? This list will help you:

Project Stars
1 minio 36,621
2 thanos 11,232
3 rook 10,539
4 juicefs 7,081
5 drive 6,469
6 upspin 6,013
7 storj 2,577
8 mc 2,306
9 s5cmd 1,397
10 heketi 1,240
11 lotusdb 1,022
12 IceFireDB 944
13 oras 787
14 fake-gcs-server 695
15 csi-digitalocean 516
16 mort 472
17 storage 408
18 directpv 328
19 kalasa 238
20 space-daemon 213
21 hackpadfs 205
22 estuary 199
23 uptoc 191
Static code analysis for 29 languages.
Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
www.sonarqube.org