Go Distributed Systems

Open-source Go projects categorized as Distributed Systems

Top 23 Go Distributed System Projects

  • GitHub repo etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: Protocol Buffers | reddit.com/r/golang | 2021-03-02
  • GitHub repo nsq

    A realtime distributed messaging platform

  • Scout

    Get performance insights in less than 4 minutes. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

  • GitHub repo go-micro

    Go Micro is a framework for distributed systems development

    Project mention: What library or framework is used to write microservices for production ? | reddit.com/r/golang | 2021-02-14
  • GitHub repo grpc-go

    The Go language implementation of gRPC. HTTP/2 based RPC

    Project mention: Go, Kafka, gRPC and MongoDB microservice with metrics and tracing 👋 | dev.to | 2021-02-28

    This article about tries to implement of clean architecture microservice using: 🚀 Kafka as messages broker gRPC Go implementation of gRPC MongoDB as database Jaeger open source, end-to-end distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus

  • GitHub repo Seaweed File System

    SeaweedFS is a distributed object store and file system to store and serve billions of files fast! Object store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

    Project mention: The Trouble with Cassandra: Why It's a Poor Choice for Object Store Metadata DB | news.ycombinator.com | 2021-02-24

    I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users.

    The article listed many known Cassandra characteristics and cited as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better.

    Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores

    I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The are about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow.

    The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap.

    So it is all depends on use cases.

  • GitHub repo micro

    Micro is a platform for cloud native development (by micro)

    Project mention: Dapr – Distributed Application Runtime | news.ycombinator.com | 2021-02-17

    Dapr's solving distributed systems development at scale. Basically as you grow your team from a handful of people to many separate teams all working on different products, the technical architecture also evolves from monolith to SOA and then microservices. It's the natural logical evolution of anything at scale. Dapr is attempting to provide the primitives you'd need to build any sort of software at scale. Usually this buckets into; authentication/authorization, configuration, data storage, pubsub messaging and a handful of other things.

    Full disclosure: I'm the author of Micro (https://github.com/micro/micro) which bakes in the same primitives but also focuses solely on development in the Go language.

  • GitHub repo NATS

    High-Performance server for NATS, the cloud native messaging system.

    Project mention: NATS, NATS Streaming & NATS JetStream + How to build a JetStream Cluster & Go Client | reddit.com/r/golang | 2021-02-28

    Here is the original conversation where it was confirmed that you can't really get a reliable distribution, even with MaxInFlight=1: https://github.com/nats-io/nats-server/issues/878 They said over time it would even out, but in my case I cannot allow a busy worker to be pushed a second message.

  • GitHub repo Nomad

    Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

    Project mention: How do i manage several processes - without containers | reddit.com/r/devops | 2021-02-25

    Nomad is a good orchestrator that's by far not so complex like kubernetes. It can not only orchestrate application containers with docker but also VMs and lxc containers.

  • GitHub repo rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Many Small Queries Are Efficient in SQLite | news.ycombinator.com | 2021-02-15
  • GitHub repo rpcx

    A zero cost, faster multi-language bidirectional microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!

  • GitHub repo git-bug

    Distributed, offline-first bug tracker embedded in git, with bridges

    Project mention: Ethical doubts on files content | reddit.com/r/ipfs | 2021-02-27

    The second reason is that I use the few spare time I have on a much more successful project: https://github.com/MichaelMure/git-bug

  • GitHub repo cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    Project mention: A 10+ year Journey | dev.to | 2020-10-21

    In 2015, Uber opened a development center in Seattle and I decided to take the leap and join the engineering team. Coincidently enough, Maxim Fateev ended up joining the Uber team in Seattle only a month after I did. At the time, Uber was running on Kafka 7 as the messaging backbone. Based on the scale they were running, they were encountering some serious operational issues. Considering Maxim and I had more than a decade worth of experience building messaging systems similar to Kafka, we decided to create the OSS project Cherami to address this Uber sized problem. After a year of working on the project, we were observing a very similar pattern to the one that we tried to solve with SWF and Durable Task Framework. When engineers needed to build with raw infrastructure primitives like queues and databases they were spending 80% of their time building resiliency into the application. This was clearly not sustainable for Uber, which was growing at an amazing pace and building a brand of "Transportation as Reliable as Running Water". This need to increase developer productivity without compromising on reliability of the system was the motivation for Maxim and I to create the OSS project Cadence. Within a very short period of time, we built a multi-tenant service hosted by our team. Cadence provided a great developer experience by enabling developers to use Golang to build and run stateful applications with very little operational overhead. Cadence grew organically within Uber and quickly became popular among developers. It slowly but surely began to emerge as the standard way to build stateful applications when reliability cannot be compromised.

  • GitHub repo raft

    Golang implementation of the Raft consensus protocol

  • GitHub repo Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

  • GitHub repo tendermint

    ⟁ Tendermint Core (BFT Consensus) in Go

    Project mention: YOLO update #1. Went all in on Cosmos on 7th Feb, traded my Uniswap and AAVE for it, everyone said it was a mistake, but I knew the cream always rises to the top and THIS CREAM is going to rise ALL THE WAY BEYOND THE MOON INTO THE COSMOS BY 18th FEB 🛸🛸🛸 | reddit.com/r/SatoshiStreetBets | 2021-02-11

    Note that they have already implemented the technology to easily build new blockchains through Tendermint https://tendermint.com

  • GitHub repo torrent

    Full-featured BitTorrent client package and utilities

  • GitHub repo KrakenD

    Ultra performant API Gateway with middlewares

  • GitHub repo dragonboat

    A feature complete and high performance multi-group Raft library in Go.

  • GitHub repo glow

    Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

  • GitHub repo juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: A distributed Posix file system built on top of Redis and S3 | news.ycombinator.com | 2021-01-12

    Most of network file system provide open-after-close consistency, same to JuiceFS, it's mentioned here [1]:

    https://github.com/juicedata/juicefs#posix-compatibility

  • GitHub repo emitter-io

    High performance, distributed and low latency publish-subscribe platform.

  • GitHub repo gleam

    Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

  • GitHub repo Dkron

    Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-03-02.

Index

What are some of the best open-source Distributed System projects in Go? This list will help you:

Project Stars
1 etcd 34,859
2 nsq 19,263
3 go-micro 15,472
4 grpc-go 13,181
5 Seaweed File System 11,506
6 micro 9,746
7 NATS 8,909
8 Nomad 8,143
9 rqlite 7,803
10 rpcx 5,451
11 git-bug 4,927
12 cadence 4,696
13 raft 4,359
14 Jocko 4,154
15 tendermint 3,963
16 torrent 3,796
17 KrakenD 3,616
18 dragonboat 3,500
19 glow 2,931
20 juicefs 2,910
21 emitter-io 2,760
22 gleam 2,754
23 Dkron 2,575