Go Distributed Systems

Open-source Go projects categorized as Distributed Systems

Top 23 Go Distributed System Projects

  • etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: Transitioning from more traditional OOP like C# to Go, what are the biggest coding style differences. | /r/golang | 2023-11-16

    Reading the standard library will give you ideas/insight about various Go idiomatic patterns/approaches, and you can see a full website/API implementation in the pkg.go.dev repository (https://github.com/golang/pkgsite). Projects like https://github.com/etcd-io/etcd may be interesting too.

  • nsq

    A realtime distributed messaging platform

    Project mention: MQTT vs. Kafka: An IoT Advocate's Perspective | news.ycombinator.com | 2023-03-14

    Interesting. What are you thoughts on NSQ?


    Was looking at it earlier today, but haven't ever tried it out.

  • InfluxDB

    Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.

  • go-micro

    A Go microservices framework

    Project mention: Ask HN: What are some examples of cloud lock-in? | news.ycombinator.com | 2022-12-05

    Had similar goals. Started by writing Go interfaces for it with Go Micro - https://go-micro.dev then opted for the platform service model as you mentioned with Micro - https://micro.dev

    I think whether it's Dapr, Micro or something else, the platform service model with well defined interfaces is the way to go. I don't think a lot of people get this yet so it's still going to be a few years before it takes off.

  • grpc-go

    The Go language implementation of gRPC. HTTP/2 based RPC

    Project mention: Full Stack Forays with Go and gRPC | dev.to | 2023-06-05

    First, I started with gRPC’s recommended starter repository for learning gRPC, their **helloworld **example, which is a part of the official gRPC repository.

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    Project mention: Show HN: OpenSign – The open source alternative to DocuSign | news.ycombinator.com | 2023-10-28

    > Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.

    Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing

    I've actually been looking for more open alternatives, but haven't found much.

    Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)

    There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/

    The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs

  • rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Adding new database engine support | /r/stalwartlabs | 2023-11-25

    I found simple distributed RQlite https://github.com/rqlite/rqlite based on raft and sqlite. How hard is to add it?

  • Nomad

    Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

    Project mention: Google Kubernetes Engine incident spanning 9 days | news.ycombinator.com | 2023-10-10
  • Onboard AI

    Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.

  • NATS

    High-Performance server for NATS.io, the cloud and edge native messaging system.

    Project mention: Scripting with NATS.io support | /r/devops | 2023-10-30

    require nats.io

  • micro

    API first development platform (by micro)

    Project mention: Show HN: Mu – A community first app platform proposal | news.ycombinator.com | 2023-06-13

    Hi HN

    I'm Asim, an engineer and author of the open source project Micro (https://micro.dev). I've been on this forum in various forms since 2009. In that time I've seen a lot of shifts in the technology landscape, participated in a few and even tried to start a company myself (which ended up VC funded). In typical HN fashion, I drank the kool-aid and applied to YC around 8-10 times. I got so far as going to mountain view for an interview but never got in. I played right into the idea of starting a fast growing company and "changing the world" by solving my own problems in the developer ecosystem but somewhere along the way lost sight of the problem I was solving. I think it speaks to my own need to seek validation (in the wrong place) but also the general nature of the ecosystem that became all about the pursuit of more funding, growth, users, etc. It really feels like while there's 70-80% value created, the other 20-30% is doing us a lot of harm. Consumer software has become addictive, corporations are becoming huge profit seeking blackholes that mostly cater to their shareholders and we have yet to see any great alternative to the existing tools and services we're now beholden too.

    A couple weeks ago I posted Micro Chat (https://micro.mu) on HN - https://news.ycombinator.com/item?id=36135683. This was my first effort in trying to solve the problems mentioned above. But I don't think it's enough. I think what we're talking about is something much bigger. And it's not going to be solved by open source, federation or a single tool. I'm starting to think we need to build something bigger, an entirely new community led app platform.

    So here's my idea. Mu (pronounced mew) is a proposal to build a new community app platform from the ground up. It would address all the problems we seem to mention here about ownership of services, addictive scrolling, profit seeking agendas and do it by empowering a community to crowdfund the creation of an alternative system that focuses on being a utility for the people. It would generally be open source, potentially self hostable, but the larger goal would be to empower a handful of people to host and run the service for everyone else, a built like how Let's Encrypt is run today.

    I've thrown this idea around a lot, started many discussions here and even tried to push a few things out but never really gotten anywhere with it. I think part of it is to just, build the smallest piece and see where it goes, hence Micro Chat, but the other part is to gauge the sentiment in the room and see whether we can actually fund something much bigger totally aligned with the community. I'm a huge fan of Google services and the utility they offer, but the nature of technology has veered so far into addiction with scrolling feeds, streaming and social media that I feel there's a need to do something now.

    If there's interest, please comment and provide feedback.


  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: A Deep Dive into the Design of Directory Quotas in JuiceFS | dev.to | 2023-10-27

    If you have any questions or would like to learn more, feel free to join discussions about JuiceFS on GitHub and the JuiceFS community on Slack.

  • temporal

    Temporal service

    Project mention: temporal VS laravel-workflow - a user suggested alternative | libhunt.com/r/temporal | 2023-08-23
  • git-bug

    Distributed, offline-first bug tracker embedded in git, with bridges

    Project mention: Gothub: Alternative front-end for GitHub written with Go | news.ycombinator.com | 2023-09-01

    Neither do the issues support. But there is git-bug [0].

    [0]: https://github.com/MichaelMure/git-bug

  • rpcx

    Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱! build for cloud!

    Project mention: Ask HN: Cool side project you have written using Golang | news.ycombinator.com | 2023-02-02

    I have the microservices framework https://rpcx.io, which is used by many companies.

  • raft

    Golang implementation of the Raft consensus protocol

    Project mention: Leader election library | /r/golang | 2023-07-07

    Depending on your exact needs, you could try HashiCorp's Raft implementation: https://github.com/hashicorp/raft

  • cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    Project mention: Ask HN: Who is hiring? (December 2023) | news.ycombinator.com | 2023-12-01

    Uber | Software Engineers | Hybrid (Denmark) | https://www.uber.com/dk/en/careers/locations/aarhus/

    Work with an amazing team responsible for the infrastructure software that makes Uber’s data centers around the world reliable and scalable. If you want to solve the toughest engineering challenges alongside some of the smartest people in the industry, Uber Aarhus is the right place for you.

    The team in Aarhus build and operate the stateless and stateful compute platforms used by nearly every other engineer in the company (Up - https://www.uber.com/en-GB/blog/up-portable-microservices-re... and Odin - https://www.uber.com/en-GB/blog/how-uber-optimized-cassandra...) as well as other related infrastructure projects such as Cadence - https://github.com/uber/cadence.

  • kitex

    Go RPC framework with high-performance and strong-extensibility for building micro-services.

  • pachyderm

    Data-Centric Pipelines and Data Versioning

    Project mention: Show HN: We scaled Git to support 1 TB repos | news.ycombinator.com | 2022-12-13

    There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.

    If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)

    The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.

    Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)

  • KrakenD

    Ultra performant API Gateway with middlewares. A project hosted at The Linux Foundation

  • easegress

    A Cloud Native traffic orchestration system

    Project mention: Thoughts on new tool - easegress - 5k stars in 8 months | /r/kubernetes | 2023-03-03

    Since there is no discussion on reddit, wanted to get people's thoughts on a relatively new project easegress which looks to have gained a lot of stars. Seems interesting and something that can't easily be compared to any other individual tool since it combines a lot of features in one.

  • tendermint

    ⟁ Tendermint Core (BFT Consensus) in Go

    Project mention: There aren't that many uses for blockchains | /r/programming | 2023-03-16

    There are good use-cases, but to much focus on cryptocurrencies. Frameworks are not well designed to be used in other use-cases. Best that I have found that doesn't forces you to a specific architecture is Tendermint, but is not even close to state-of-the-art performance and scalability. Everything else is designed around cryptocurrency and smart contracts.

  • gaia

    Build powerful pipelines in any programming language.

  • torrent

    Full-featured BitTorrent client package and utilities

    Project mention: 0x0: Share Files from Terminal | news.ycombinator.com | 2023-04-08

    https://github.com/anacrolix/torrent/blob/master/cmd/torrent... does exactly that. Install with `go get github.com/anacrolix/torrent/cmd/torrent@latest`, and then run `torrent serve `.

  • Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-12-01.

Go Distributed Systems related posts


What are some of the best open-source Distributed System projects in Go? This list will help you:

Project Stars
1 etcd 45,000
2 nsq 23,949
3 go-micro 21,001
4 grpc-go 19,169
5 seaweedfs 18,752
6 rqlite 14,194
7 Nomad 14,117
8 NATS 13,900
9 micro 11,917
10 juicefs 8,898
11 temporal 8,574
12 git-bug 7,906
13 rpcx 7,790
14 raft 7,520
15 cadence 7,455
16 kitex 6,382
17 pachyderm 6,004
18 KrakenD 5,818
19 easegress 5,575
20 tendermint 5,566
21 gaia 5,124
22 torrent 4,998
23 Jocko 4,799
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives