Go Distributed Systems

Open-source Go projects categorized as Distributed Systems

Top 23 Go Distributed System Projects

  • GitHub repo etcd

    Distributed reliable key-value store for the most critical data of a distributed system

    Project mention: Cannot connect to OpenShift cluster using oc tool and admin console. | reddit.com/r/openshift | 2021-06-09

    I found this issue https://github.com/etcd-io/etcd/issues/11949

  • GitHub repo nsq

    A realtime distributed messaging platform

    Project mention: For those running Go in production at scale, what do you use for distributed task queues? | reddit.com/r/golang | 2021-06-04

    Here is link number 1 - Previous text "NSQ"

  • GitHub repo go-micro

    Go Micro is a framework for distributed systems development

    Project mention: Why and how we built Distributed with Next.js and Micro | dev.to | 2021-03-26

    Distributed is a live social chat app built as a Jamstack demo using Next.js and Micro. We built it to demonstrate the value proposition of M3O - a cloud platform for API development. This post explains what went into building Distributed in just a few weeks and how M3O helped rapidly build our MVP. You can find the source code for distributed on Github. If you want to build and host your own version signup to M3O and start running the same services from our open source repository micro/services. Why we built Distributed Distributed was built as a jamstack demo to show how you could leverage M3O as an API backend for rapid MVP development. M3O itself is a cloud platform for API development, built on the popular open source project Micro. Micro enables you to quickly build APIs in Go on the backend and M3O provides simple free hosting of those services. We wanted to show the Jamstack audience how you could quickly leverage those APIs to build something on the frontend. Not only that, we really wanted to understand and experience the frontend developers perspective through dogfooding of our own APIs rather than just throwing stuff over the wall and hoping it works. Hopefully in that we we've done is demonstrate the value of our platform and how others can also make use of it with a real world app like Distributed to learn from. Let's talk more about Jamstack now. Jamstack Development Jamstack is a new architecture pattern for frontend which offloads dynamic aspects of the stack to javascript and third party APIs. Vercel, the makers of Next.js and related companies are pioneering the way forward for jamstack development. Credit jamstack.org JAMstack stands for Javascript, API and Markup. The static part of the application is deployed to a CDN with javascript dynamically loading various pieces of dynamic content from backend APIs and rendering it. Why we chose Next.js Next.js is a massively popular react based framework for Jamstack development. When we were looking at building out a demo on top of M3O we had the choice of going down a number of routes but what really appealed to us was the how deliberate a lot of the choices were in how the Vercel team had constructed the Next.js framework. Being framework creators ourselves with the dominant framework Go Micro for Go, we could appreciate the efforts required and strong opinions needed to drive such adoption and success. Vercel has done a phenomenal job in this way. Beyond praising Vercel's efforts. The Next.js framework includes a lot of key components needed for the Jamstack including server side rendering, api routes and typescript support. For us these were mandatory feature requirements when building against not only our APIs but third party providers. Micro for the Backend Micro is an open source cloud platform for API development. With modern day complexity in writing software for the cloud, Micro has attempted to distill that down to a handful of primitives and a framework for building services in Go. Micro took learnings from the original Go Micro framework and focused on not just gRPC based service development but actually packaging together a runtime and platform which exposes those services automatically as APIs. What this means is we can write microservices on the backend using gRPC and protobuf and immediately provide value to consumers and clients on the frontend via HTTP/JSON. To learn more about that check out the project at micro.mu or the hosted platform at m3o.com. Building on Netlify We initially chose Netlify for hosting as we saw many people adopting it for Jamstack apps. Initially this proved really great for static content. As our apps got more complex and we started to build out the Distributed demo we found Netlify no longer scaled with our basic needs. The first example we can share is Netlify Functions for Next.js API routes. Next.js routes can be turned into Netlify Functions which are essentially hosted as AWS Lambda functions. It's a clever way of pushing certain requirements to the server side, like calling third party APIs with keys you don't want to expose to the client. Next.js is great in this regard and plugins like netlify-plugin-nextjs and next-on-netlify let us do this really quickly but the performance left a lot to be desired. Our APIs are primarily hosted in London on DigitalOcean and while Netlify has a CDN for static content, the Lambda functions are deployed in a single region in US-East on AWS. For those who've suffered this pain you know exactly what that means. We were making cross atlantic calls from JS in the client to api routes on lambda and then finally to our apis. Needless to say this didn't scale for us. We weren't able to reach out the Netlify team to get help and so in frustration had to go down the self hosted route. Note we did test out Vercel and found the experience to be faster but self hosting on DigitalOcean just made more sense for our demo needs. Switching to Self Hosted One of the things DigitalOcean now provides is App Platform, a container hosting solution which lets you pick regions, does TLS certificate management for your custom domain and automatic builds from Git. This turned out to be a perfect solution for self hosted Next.js apps. Next.js at the end of the day is a React and node.js based application. As much as you may want to separate out the static content to something like Netlify and functions on Lambda, it equally just makes sense to host the entire thing in one place and run many copies of it much like we did in the old php and rails days. Because the APIs are colocated with the frontend we find this experience fairly fast, sub 100ms for all the API calls but we know it's not an ideal demonstration of the Jamstack's architecture and so we'll be working towards hosting on Vercel in the future to showcase that experience. How It All Fits Together We're running Distributed as a Next.js application on the frontend talking to Micro APIs on the backend. All of this is constructed as API routes in Next.JS firing requests at our M3O platform and the various APIs we need. Let's walk through a typical request. For example, when loading a group we need to get the group info, user profile, chats and more. We could do this as a GraphQL APIbut that would require too much stitching together in terms of the schema on the backend. Instead we're using protobuf and RPC for rapid development there and Micro automagically exposes that as a HTTP/JSON API. So a typical flow is like so. Frontend makes a request to /api/groups/[id] which loads the api code in the Next.js app We validate the user is logged in by calling the /users/Validate endpoint and ff authenticated load the group data by id using /groups/Read Skipping ahead, we'll load group messages via /threads/ListConversations and private messages using /chats/ListMessages We can check for already read messages via a "seen" API and then subscribe to the streams API for instant messaging Finally we render everything based on the content loaded for the user .gist-data { max-height: 600px; overflow: auto;} Here's a code "snippet" for those interested. From an MVP standpoint this is just a very quick and rapid way for us to build against numerous separate APIs on the backend all hosted in the same place. For anyone interested in the "call" function. It's simply a small function we're using to call the Micro APIs on the backend. Remember Micro turns any RPC based service into a HTTP/JSON API automatically via an API gateway. M3O provides hosting for all this. Performance & Productivity Aside from the structural benefits of a framework like Next.js we find it really unlocks significant productivity by providing an opinionated approach to frontend development. That coupled with Micro on the backend and our APIs hosted on M3O it's enabled us to rapidly ship this MVP within the space of 4-6 weeks with mostly 1 person doing the work. That really speaks to the power of the combination of Next.js and Micro. For this demo we built APIs for user management, group messaging, websocket streaming, sending invite emails and audio/video through Twilio WebRTC. One can only imagine where it would go with a dedicated team and full product focus. On the performance side, Next.js is blazingly fast by all measures. Whether it be the local reload for development or the server side rendering. It all adds to a really snappy experience on both the development and consumption side of things. With the backend we tried to pair this with Go based APIs written with Micro to ensure not just speed of development but also speed of delivery. All in all, we think Next.js and Micro are the perfect pairing for any Jamstack and API based development. Conclusions Thanks for reading this post on how we built Distributed on the Jamstack using Next.js and Micro. Find the source code for distributed on Github. If you want to build and host your own version signup to M3O and start running the same services from our open source repository micro/services. Reach out on slack or twitter if you have any questions.

  • GitHub repo grpc-go

    The Go language implementation of gRPC. HTTP/2 based RPC

    Project mention: Killing PubSub connection when client of server side stream dies in go | reddit.com/r/golang | 2021-04-30
  • GitHub repo Seaweed File System

    SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

    Project mention: Three node system needs low latency distributed storage | reddit.com/r/homelab | 2021-06-17

    SeaweedFS has been bookmarked for a while but I haven't had a chance to play with it. But 100ms between nodes is an architecture problem, not a software problem (IMO).

  • GitHub repo micro

    Micro is a distributed OS built for the Cloud (by micro)

    Project mention: Micro - A platform for microservices development | reddit.com/r/microservices | 2021-04-16
  • GitHub repo NATS

    High-Performance server for NATS.io, the cloud and edge native messaging system.

    Project mention: [HIRING] Senior Backend Developer (Golang) at Valocode (Helsinki, Finland) | Salary: 60k-70k EUR - depending on equity package | reddit.com/r/RemoteJobs | 2021-05-31

    About the role: You will have a crucial role in developing the Bubbly backend which is entirely written in Golang. We are using NATS (https://nats.io) and a service-oriented architecture (not quite microservice) where the services communicate via NATS message queues. Postgres is the primary database right now, but we have plans for immutable, append-only databases in the future for more compliance-related activities. As the team was founded from a DevOps company, Verifa, we enjoy our DevOps practices and would hope you will enjoy it also.

  • GitHub repo go-zero

    go-zero is a web and rpc framework written in Go. It's born to ensure the stability of the busy sites with resilient design. Builtin goctl greatly improves the development productivity.

    Project mention: Any idea on developing an IDL to simplify the restful API development? | news.ycombinator.com | 2021-05-31
  • GitHub repo Nomad

    Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.

    Project mention: Replit used legal threats to kill my open-source project | news.ycombinator.com | 2021-06-07

    I would look at how the major clouds are running their compute instances. This might help:

    https://www.nomadproject.io/

    I think you will end up needing VMs. I have been thinking of using Tekton to make a CI/CD service but I would pretty much need to smash the whole k8s VM/Node every time I do something and only allow one participant at a time.

  • GitHub repo rqlite

    The lightweight, distributed relational database built on SQLite

    Project mention: Comparing rqlite with tidb and any other distributed sql server alternatives? | reddit.com/r/golang | 2021-06-10

    Does anyone know production deployment of rqlite, how stable it is, and how well does it scale?

  • GitHub repo rpcx

    Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!

  • GitHub repo git-bug

    Distributed, offline-first bug tracker embedded in git, with bridges

    Project mention: Go is powering enterprise developers: Developer survey results | reddit.com/r/golang | 2021-04-23

    It's my general experience working on various/complex software and with talented people. For the more go specific part, writing https://github.com/MichaelMure/git-bug.

  • GitHub repo cadence

    Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

    Project mention: For those running Go in production at scale, what do you use for distributed task queues? | reddit.com/r/golang | 2021-06-04

    We use cadence https://github.com/uber/cadence

  • GitHub repo raft

    Golang implementation of the Raft consensus protocol

  • GitHub repo Jocko

    Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

    Project mention: ZooKeeper-free Kafka is out. First Demo | news.ycombinator.com | 2021-03-30

    Reminded me of this project: https://github.com/travisjeffery/jocko

    Kafka implemented in Go without needing Zookeeper.

  • GitHub repo KrakenD

    Ultra performant API Gateway with middlewares. A project hosted at The Linux Foundation

    Project mention: Authelia is an open-source authentication/authorization server with 2FA/SSO | news.ycombinator.com | 2021-03-10

    Thanks for the pointer to Express API Gateway.

    A took a look at Krakend a while back, and it didn't seem to support it either [1]

    [1] https://github.com/devopsfaith/krakend/issues/274

  • GitHub repo tendermint

    ⟁ Tendermint Core (BFT Consensus) in Go

    Project mention: EIP1559 is going live soon. Here’s what it means for Ethereum. | reddit.com/r/CryptoCurrency | 2021-06-15
  • GitHub repo torrent

    Full-featured BitTorrent client package and utilities

    Project mention: Getting torrent file hash | reddit.com/r/golang | 2021-04-27

    I haven't vetted this library, but https://github.com/anacrolix/torrent looks useful.

  • GitHub repo dragonboat

    A feature complete and high performance multi-group Raft library in Go.

  • GitHub repo juicefs

    JuiceFS is a distributed POSIX file system built on top of Redis and S3.

    Project mention: "JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environment." | reddit.com/r/programming | 2021-04-20
  • GitHub repo glow

    Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant. (by chrislusf)

  • GitHub repo emitter-io

    High performance, distributed and low latency publish-subscribe platform.

  • GitHub repo gleam

    Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly. (by chrislusf)

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-06-17.

Index

What are some of the best open-source Distributed System projects in Go? This list will help you:

Project Stars
1 etcd 36,221
2 nsq 19,753
3 go-micro 16,133
4 grpc-go 13,889
5 Seaweed File System 12,177
6 micro 10,139
7 NATS 9,466
8 go-zero 8,874
9 Nomad 8,643
10 rqlite 8,488
11 rpcx 5,725
12 git-bug 5,207
13 cadence 5,015
14 raft 4,797
15 Jocko 4,287
16 KrakenD 4,222
17 tendermint 4,157
18 torrent 3,910
19 dragonboat 3,672
20 juicefs 3,308
21 glow 2,934
22 emitter-io 2,886
23 gleam 2,829
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com