Top 23 Go Distributed System Projects
Distributed reliable key-value store for the most critical data of a distributed systemProject mention: Protocol Buffers | reddit.com/r/golang | 2021-03-02
A realtime distributed messaging platform
Get performance insights in less than 4 minutes. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
Go Micro is a framework for distributed systems developmentProject mention: What library or framework is used to write microservices for production ? | reddit.com/r/golang | 2021-02-14
The Go language implementation of gRPC. HTTP/2 based RPCProject mention: Go, Kafka, gRPC and MongoDB microservice with metrics and tracing 👋 | dev.to | 2021-02-28
This article about tries to implement of clean architecture microservice using: 🚀 Kafka as messages broker gRPC Go implementation of gRPC MongoDB as database Jaeger open source, end-to-end distributed tracing Prometheus monitoring and alerting Grafana for to compose observability dashboards with everything from Prometheus
SeaweedFS is a distributed object store and file system to store and serve billions of files fast! Object store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.Project mention: The Trouble with Cassandra: Why It's a Poor Choice for Object Store Metadata DB | news.ycombinator.com | 2021-02-24
I am working on SeaweedFS, which supports S3 API for object store, and can also use Cassandra as the metadata db. Cassandra has been performing well for most SeaweedFS users.
The article listed many known Cassandra characteristics and cited as limitations. However, it all depends on use cases. There are no file system that works for all cases, and not all of them needs ACID, CA vs CP, etc. The rest points are not convincing either. They are related to how to design the data structure better.
Actually, SeaweedFS can use many other database/KV stores as the metadata DB. The list includes Redis, Cassandra, HBase, MySql, Postgres, Etcd, ElasticSearch, etc. https://github.com/chrislusf/seaweedfs/wiki/Filer-Stores
I did find one drawback for Cassandra as the metadata store though. One use case is that the customer uploaded a lot of zip files to one folder /tmp, unzip them, and then moved to a final folder. The are about 3000 files per second created and then deleted. Being a LSM structure, the tombstones quickly pile up and the directory listing was slow.
The solution was to use Redis for that /tmp folder, and still use Cassandra for the rest of folders. With Redis B-tree structure, the creation and deletion are cheap.
So it is all depends on use cases.
Micro is a platform for cloud native development (by micro)Project mention: Dapr – Distributed Application Runtime | news.ycombinator.com | 2021-02-17
Dapr's solving distributed systems development at scale. Basically as you grow your team from a handful of people to many separate teams all working on different products, the technical architecture also evolves from monolith to SOA and then microservices. It's the natural logical evolution of anything at scale. Dapr is attempting to provide the primitives you'd need to build any sort of software at scale. Usually this buckets into; authentication/authorization, configuration, data storage, pubsub messaging and a handful of other things.
Full disclosure: I'm the author of Micro (https://github.com/micro/micro) which bakes in the same primitives but also focuses solely on development in the Go language.
High-Performance server for NATS, the cloud native messaging system.Project mention: NATS, NATS Streaming & NATS JetStream + How to build a JetStream Cluster & Go Client | reddit.com/r/golang | 2021-02-28
Here is the original conversation where it was confirmed that you can't really get a reliable distribution, even with MaxInFlight=1: https://github.com/nats-io/nats-server/issues/878 They said over time it would even out, but in my case I cannot allow a busy worker to be pushed a second message.
Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.Project mention: How do i manage several processes - without containers | reddit.com/r/devops | 2021-02-25
Nomad is a good orchestrator that's by far not so complex like kubernetes. It can not only orchestrate application containers with docker but also VMs and lxc containers.
The lightweight, distributed relational database built on SQLiteProject mention: Many Small Queries Are Efficient in SQLite | news.ycombinator.com | 2021-02-15
A zero cost, faster multi-language bidirectional microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱!
Distributed, offline-first bug tracker embedded in git, with bridgesProject mention: Ethical doubts on files content | reddit.com/r/ipfs | 2021-02-27
The second reason is that I use the few spare time I have on a much more successful project: https://github.com/MichaelMure/git-bug
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.Project mention: A 10+ year Journey | dev.to | 2020-10-21
In 2015, Uber opened a development center in Seattle and I decided to take the leap and join the engineering team. Coincidently enough, Maxim Fateev ended up joining the Uber team in Seattle only a month after I did. At the time, Uber was running on Kafka 7 as the messaging backbone. Based on the scale they were running, they were encountering some serious operational issues. Considering Maxim and I had more than a decade worth of experience building messaging systems similar to Kafka, we decided to create the OSS project Cherami to address this Uber sized problem. After a year of working on the project, we were observing a very similar pattern to the one that we tried to solve with SWF and Durable Task Framework. When engineers needed to build with raw infrastructure primitives like queues and databases they were spending 80% of their time building resiliency into the application. This was clearly not sustainable for Uber, which was growing at an amazing pace and building a brand of "Transportation as Reliable as Running Water". This need to increase developer productivity without compromising on reliability of the system was the motivation for Maxim and I to create the OSS project Cadence. Within a very short period of time, we built a multi-tenant service hosted by our team. Cadence provided a great developer experience by enabling developers to use Golang to build and run stateful applications with very little operational overhead. Cadence grew organically within Uber and quickly became popular among developers. It slowly but surely began to emerge as the standard way to build stateful applications when reliability cannot be compromised.
Golang implementation of the Raft consensus protocol
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
⟁ Tendermint Core (BFT Consensus) in GoProject mention: YOLO update #1. Went all in on Cosmos on 7th Feb, traded my Uniswap and AAVE for it, everyone said it was a mistake, but I knew the cream always rises to the top and THIS CREAM is going to rise ALL THE WAY BEYOND THE MOON INTO THE COSMOS BY 18th FEB 🛸🛸🛸 | reddit.com/r/SatoshiStreetBets | 2021-02-11
Note that they have already implemented the technology to easily build new blockchains through Tendermint https://tendermint.com
Full-featured BitTorrent client package and utilities
Ultra performant API Gateway with middlewares
A feature complete and high performance multi-group Raft library in Go.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.Project mention: A distributed Posix file system built on top of Redis and S3 | news.ycombinator.com | 2021-01-12
Most of network file system provide open-after-close consistency, same to JuiceFS, it's mentioned here :
High performance, distributed and low latency publish-subscribe platform.
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io
What are some of the best open-source Distributed System projects in Go? This list will help you:
|5||Seaweed File System||11,506|