Go Distributed Systems

Open-source Go projects categorized as Distributed Systems

Top 23 Go Distributed System Projects

  • etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: A poor man's API | dev.to | 2022-11-23

    APISIX stores its configuration in etcd

  • nsq

    A realtime distributed messaging platform

    Project mention: Infinite loop pattern to poll for a queue in a REST server app | reddit.com/r/node | 2022-10-16

    Queue consumers are interesting because there are many solutions for them, from using Redis and persisting the data in a data store - but for fast and scalable the approach I would take is something like SQS (as I advocate AWS even free tier) or NSQ for managing your own distributed producers and consumers.

  • Zigi

    Workflow assistant built for devs & their teams. Automate the mundane part of your day, with live actionable messages for your GitHub & Jira tasks.

  • go-micro

    A Go microservices framework

    Project mention: Go Micro: a standard library for distributed systems development | news.ycombinator.com | 2022-09-30
  • grpc-go

    The Go language implementation of gRPC. HTTP/2 based RPC

    Project mention: Leveling Up Reddit's Core - The Transition from Thrift to gRPC | reddit.com/r/RedditEng | 2022-09-19

    gRPCs encoding is robust enough to allow us to easily support alternative serialization formats through something called content-subtype. By defining a content-subtype, we can register a codec that will be able to perform serialization of thrift models. This was the key insight that allowed us to decouple the model refactor from the stub migration. Looking to the future, this also means that we will be able to provide a protobuf-based version of the same API, giving users a path forward for migrating away from Thrift models.

  • seaweedfs

    SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.

    Project mention: Can you setup Gluster or Ceph to scale from ONE node ( Data Replication)? | reddit.com/r/homelab | 2022-11-10

    You could maybe get seaweedfs to do something like this. (and run it in k8s)

  • Nomad

    Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

    Project mention: What is the best solution to orchestrate Docker containers ? | reddit.com/r/docker | 2022-11-11

    I'd like to also mention hashicorp nomad + consul. They're worth evaluating.

  • rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Crsql – Multi-writer and CRDT support for SQLite | news.ycombinator.com | 2022-11-14
  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • NATS

    High-Performance server for NATS.io, the cloud and edge native messaging system.

    Project mention: Senior developers, how did you start and how to improve? | reddit.com/r/learnprogramming | 2022-11-09

    Write a client server chat app using a bus like https://nats.io to communicate between them. It's easy, you can then Dockerise them later and run it via docker-compose. You can even use this to write something similar to https://temp.chat

  • micro

    An API first development platform (by micro)

    Project mention: Micro - An API first development platform | reddit.com/r/microservices | 2022-11-15
  • rpcx

    Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱! build for cloud!

    Project mention: Distributed IM Service in Golang | dev.to | 2022-04-19

    rpcx : High-performance, feature-rich microservices framework

  • juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: POSIX Compatibility Comparison among four file system on the cloud | dev.to | 2022-11-21

    From Juicedata/JuiceFS ! (0ᴗ0✿)

  • git-bug

    Distributed, offline-first bug tracker embedded in git, with bridges

    Project mention: Coping strategies for the serial project hoarder | news.ycombinator.com | 2022-11-28

    (Disclaimer: I work for GitHub, but not on issues, and I've only been here 6 months)

    The counterpoint to this is that GitHub has been around for a long time and I don't think all of these free features are going anywhere. Even beyond that, there's an API you can use to get your data out and lots of tools build on that API.

    https://github.com/MichaelMure/git-bug came up here recently and that looks pretty good, though I haven't tried it. I suspect the usability of a lot of these tools will lag behind GitHub Issues/Projects.

  • raft

    Golang implementation of the Raft consensus protocol

    Project mention: Raft Is So Fetch: The Raft Consensus Algorithm Explained Through Mean Girls | news.ycombinator.com | 2022-10-03
  • cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    Project mention: [P] My co-founder and I quit our engineering jobs at AWS to build “Tensor Search”. Here is why. | reddit.com/r/MachineLearning | 2022-09-21

    Emit events from your primary DB (postgres, etc.) to something like kafka or rabbitmq and then catch that in your search engine. There's also some end-to-end solutions like temporal (temporal.io) or cadence (https://cadenceworkflow.io/)

  • pachyderm

    Data-Centric Pipelines and Data Versioning

    Project mention: Awesome list of VCs investing in commercial open-source startups | reddit.com/r/opensource | 2022-09-14

    Pachyderm - License prevents competition.

  • temporal

    Temporal service

    Project mention: Temporal PHP SDK: Scalable and resilent workflow orchestration on PHP | reddit.com/r/PHP | 2022-11-15

    Documentation

  • kitex

    Go RPC framework with high-performance and strong-extensibility for building micro-services.

  • KrakenD

    Ultra performant API Gateway with middlewares. A project hosted at The Linux Foundation

    Project mention: Share Your Code.. Share your most unique piece of Go code. | reddit.com/r/golang | 2022-10-15

    KrakenD and Lura https://github.com/krakendio/krakend-ce https://github.com/luraproject/lura

  • tendermint

    ⟁ Tendermint Core (BFT Consensus) in Go

    Project mention: How to Become a BlockX Validators. What all thing you should keep in Mind. | reddit.com/r/BlockX | 2022-11-11

    BlockX is based on Tendermint Core, which relies on a set of validators that are responsible for committing new blocks in the blockchain. These validators participate in the consensus protocol by broadcasting votes which contain cryptographic signatures signed by each validator's private key.

  • easegress

    A Cloud Native traffic orchestration system

    Project mention: Kubernetes Ingress: Nginx Ingress Edition | dev.to | 2022-05-04

    Easegress IngressController is an Easegress based API gateway that can run as an ingress controller.

  • gaia

    Build powerful pipelines in any programming language.

  • Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

    Project mention: Ask HN: What interesting problems are you working on? ( 2022 Edition) | news.ycombinator.com | 2022-09-16

    Still early stages but building on top of this. - https://github.com/travisjeffery/jocko

  • torrent

    Full-featured BitTorrent client package and utilities

    Project mention: Full-featured BitTorrent client package and utilities for Go | news.ycombinator.com | 2022-11-06
  • Scout APM

    Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-11-28.

Go Distributed Systems related posts

Index

What are some of the best open-source Distributed System projects in Go? This list will help you:

Project Stars
1 etcd 41,906
2 nsq 22,943
3 go-micro 19,701
4 grpc-go 17,149
5 seaweedfs 15,921
6 Nomad 12,845
7 rqlite 12,467
8 NATS 11,873
9 micro 11,481
10 rpcx 7,338
11 juicefs 7,047
12 git-bug 6,778
13 raft 6,530
14 cadence 6,437
15 pachyderm 5,693
16 temporal 5,677
17 kitex 5,321
18 KrakenD 5,315
19 tendermint 5,281
20 easegress 4,888
21 gaia 4,856
22 Jocko 4,682
23 torrent 4,564
Build time-series-based applications quickly and at scale.
InfluxDB is the Time Series Data Platform where developers build real-time applications for analytics, IoT and cloud-native services in less time with less code.
www.influxdata.com