Go Distributed Systems

Open-source Go projects categorized as Distributed Systems

Top 23 Go Distributed System Projects

  • etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Latest mention: Gopher Gold #16 - Wed Oct 21 2020 | dev.to | 2020-10-21

    etcd-io/etcd (Go): Distributed reliable key-value store for the most critical data of a distributed system

  • nsq

    A realtime distributed messaging platform

  • go-micro

    Go Micro is a framework for distributed systems development

    Latest mention: Gopher Gold #16 - Wed Oct 21 2020 | dev.to | 2020-10-21

    asim/go-micro (Go): A Go microservices development framework

  • go-micro

    Go Micro is a framework for distributed systems development

    Latest mention: Tturning dep project into go modules | reddit.com/r/golang | 2021-01-11

    after you said that, i changed to github..com/micro/go-micro/v2 all the packages that use go-micro repo, but i keep receiving the same errors. What does "go: converting Gopkg.lock: stat github.com/micro/go-micro/v2/[email protected]: github.com/micro/[email protected]: invalid version: go.mod has post-v1 module path "github.com/micro/go-micro/v2" at revision eb1e22bd10b0" mean? Why it says "github.com/micro/[email protected]" if i set to version 2.9.1?

  • grpc-go

    The Go language implementation of gRPC. HTTP/2 based RPC

    Latest mention: protoc plugin, with support for google/api/annotations.proto | reddit.com/r/golang | 2021-01-22
  • Seaweed File System

    SeaweedFS is a distributed object store and file system to store and serve billions of files fast! Object store has O(1) disk seek, transparent cloud integration. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

    Latest mention: I received first-ever donation on my open-source side project and it felt great | news.ycombinator.com | 2021-01-12

    I am receiving $337 per month as of now for https://github.com/chrislusf/seaweedfs

    Not something to be proud of if including the time spent to evolve the project.

  • micro

    Micro is a platform for cloud native development

    Latest mention: Show HN: Micro – A platform for cloud native development in Go | news.ycombinator.com | 2021-01-25

    The license[0] for this project might be problematic. It is worded very vague and far reaching.

    [0]: https://github.com/micro/micro/blob/master/LICENSE

  • NATS

    High-Performance server for NATS, the cloud native messaging system.

    Latest mention: Tiny message broker written in Go - nothing serious. Born out of COVID-boredom. Would love your suggestions. | reddit.com/r/golang | 2020-12-23

    NATS has it's own custom text based protocol, and the client ecosystem is not as mature as others', like MQTT. In fact, I developed the NATS Dart client initially to be used from Flutter apps, but now can be used on Dart servers as well. Ivan from NATS is currently working on native MQTT v3.5 support (releasing sometime soon) and v5 support landing in Q1 2021 (Source: NATS Roadmap). I see here that they're building the Go implementation from scratch, which is interesting. They also seem to be converting every MQTT topic to a NATS topic for backward compatibility (obviously, duh) - I'm not sure if this brings any challenges down the line.

  • Nomad

    Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

    Latest mention: Gopher Gold #14 - Wed Oct 07 2020 | dev.to | 2020-10-07

    hashicorp/nomad (Go): Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

  • rqlite

    The lightweight, distributed relational database built on SQLite.

    Latest mention: rqlite/rqlite | reddit.com/r/devopsish | 2021-01-23
  • rpcx

    A zero cost, faster multi-language bidirectional microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!

  • git-bug

    Distributed, offline-first bug tracker embedded in git, with bridges

    Latest mention: Anyone using Fossil or Pijul with Rust projects? | reddit.com/r/rust | 2021-01-22

    (Optionally with something like git-bug that lets you sync issues with GitHub, GitLab, etc.)

  • cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    Latest mention: A 10+ year Journey | dev.to | 2020-10-21

    In 2015, Uber opened a development center in Seattle and I decided to take the leap and join the engineering team. Coincidently enough, Maxim Fateev ended up joining the Uber team in Seattle only a month after I did. At the time, Uber was running on Kafka 7 as the messaging backbone. Based on the scale they were running, they were encountering some serious operational issues. Considering Maxim and I had more than a decade worth of experience building messaging systems similar to Kafka, we decided to create the OSS project Cherami to address this Uber sized problem. After a year of working on the project, we were observing a very similar pattern to the one that we tried to solve with SWF and Durable Task Framework. When engineers needed to build with raw infrastructure primitives like queues and databases they were spending 80% of their time building resiliency into the application. This was clearly not sustainable for Uber, which was growing at an amazing pace and building a brand of "Transportation as Reliable as Running Water". This need to increase developer productivity without compromising on reliability of the system was the motivation for Maxim and I to create the OSS project Cadence. Within a very short period of time, we built a multi-tenant service hosted by our team. Cadence provided a great developer experience by enabling developers to use Golang to build and run stateful applications with very little operational overhead. Cadence grew organically within Uber and quickly became popular among developers. It slowly but surely began to emerge as the standard way to build stateful applications when reliability cannot be compromised.

  • raft

    Golang implementation of the Raft consensus protocol

  • Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

  • tendermint

    ⟁ Tendermint Core (BFT Consensus) in Go

  • torrent

    Full-featured BitTorrent client package and utilities

  • KrakenD

    Ultra performant API Gateway with middlewares

  • dragonboat

    A feature complete and high performance multi-group Raft library in Go.

  • glow

    Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

  • gleam

    Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

  • emitter-io

    High performance, distributed and low latency publish-subscribe platform.

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Latest mention: A distributed Posix file system built on top of Redis and S3 | news.ycombinator.com | 2021-01-12

    Most of network file system provide open-after-close consistency, same to JuiceFS, it's mentioned here [1]:

    https://github.com/juicedata/juicefs#posix-compatibility

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-01-25.

Index

What are some of the best open-source Distributed System projects in Go? This list will help you:

Project Stars
1 etcd 34,514
2 nsq 19,105
3 go-micro 15,279
4 go-micro 15,270
5 grpc-go 13,011
6 Seaweed File System 11,340
7 micro 9,537
8 NATS 8,776
9 Nomad 7,898
10 rqlite 7,367
11 rpcx 5,392
12 git-bug 4,896
13 cadence 4,614
14 raft 4,262
15 Jocko 4,118
16 tendermint 3,874
17 torrent 3,766
18 KrakenD 3,486
19 dragonboat 3,454
20 glow 2,915
21 gleam 2,731
22 emitter-io 2,706
23 juicefs 2,634