SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Go Distributed System Projects
Distributed reliable key-value store for the most critical data of a distributed systemProject mention: Transitioning from more traditional OOP like C# to Go, what are the biggest coding style differences. | /r/golang | 2023-11-16
Reading the standard library will give you ideas/insight about various Go idiomatic patterns/approaches, and you can see a full website/API implementation in the pkg.go.dev repository (https://github.com/golang/pkgsite). Projects like https://github.com/etcd-io/etcd may be interesting too.
A realtime distributed messaging platform
Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
A Go microservices frameworkProject mention: Ask HN: What are some examples of cloud lock-in? | news.ycombinator.com | 2022-12-05
I think whether it's Dapr, Micro or something else, the platform service model with well defined interfaces is the way to go. I don't think a lot of people get this yet so it's still going to be a few years before it takes off.
The Go language implementation of gRPC. HTTP/2 based RPCProject mention: Full Stack Forays with Go and gRPC | dev.to | 2023-06-05
First, I started with gRPC’s recommended starter repository for learning gRPC, their **helloworld **example, which is a part of the official gRPC repository.
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.Project mention: Show HN: OpenSign – The open source alternative to DocuSign | news.ycombinator.com | 2023-10-28
> Theoretically they could swap with minio but last time we used it it was not a drop-in replacement yet.
Depends on whether AGPL v3 works for you or not (or whether you decide to pay them), I guess: https://min.io/pricing
I've actually been looking for more open alternatives, but haven't found much.
Zenko CloudServer seemed to be somewhat promising, but doesn't seem to be managed very actively: https://github.com/scality/cloudserver/issues/4986 (their Docker images on DockerHub were last updated 10 months ago, which is what the homepage links to; blog doesn't seem active since 2019, forums don't have much going on, despite some action on GitHub still)
There was also Garage, but that one is also AGPL v3: https://garagehq.deuxfleurs.fr/
The closest I got was discovering that SeaweedFS has an S3 compatible mode: https://github.com/seaweedfs/seaweedfs
The lightweight, distributed relational database built on SQLiteProject mention: Adding new database engine support | /r/stalwartlabs | 2023-11-25
I found simple distributed RQlite https://github.com/rqlite/rqlite based on raft and sqlite. How hard is to add it?
Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.Project mention: Google Kubernetes Engine incident spanning 9 days | news.ycombinator.com | 2023-10-10
Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.
High-Performance server for NATS.io, the cloud and edge native messaging system.Project mention: Scripting with NATS.io support | /r/devops | 2023-10-30
API first development platform (by micro)Project mention: Show HN: Mu – A community first app platform proposal | news.ycombinator.com | 2023-06-13
I'm Asim, an engineer and author of the open source project Micro (https://micro.dev). I've been on this forum in various forms since 2009. In that time I've seen a lot of shifts in the technology landscape, participated in a few and even tried to start a company myself (which ended up VC funded). In typical HN fashion, I drank the kool-aid and applied to YC around 8-10 times. I got so far as going to mountain view for an interview but never got in. I played right into the idea of starting a fast growing company and "changing the world" by solving my own problems in the developer ecosystem but somewhere along the way lost sight of the problem I was solving. I think it speaks to my own need to seek validation (in the wrong place) but also the general nature of the ecosystem that became all about the pursuit of more funding, growth, users, etc. It really feels like while there's 70-80% value created, the other 20-30% is doing us a lot of harm. Consumer software has become addictive, corporations are becoming huge profit seeking blackholes that mostly cater to their shareholders and we have yet to see any great alternative to the existing tools and services we're now beholden too.
A couple weeks ago I posted Micro Chat (https://micro.mu) on HN - https://news.ycombinator.com/item?id=36135683. This was my first effort in trying to solve the problems mentioned above. But I don't think it's enough. I think what we're talking about is something much bigger. And it's not going to be solved by open source, federation or a single tool. I'm starting to think we need to build something bigger, an entirely new community led app platform.
So here's my idea. Mu (pronounced mew) is a proposal to build a new community app platform from the ground up. It would address all the problems we seem to mention here about ownership of services, addictive scrolling, profit seeking agendas and do it by empowering a community to crowdfund the creation of an alternative system that focuses on being a utility for the people. It would generally be open source, potentially self hostable, but the larger goal would be to empower a handful of people to host and run the service for everyone else, a built like how Let's Encrypt is run today.
I've thrown this idea around a lot, started many discussions here and even tried to push a few things out but never really gotten anywhere with it. I think part of it is to just, build the smallest piece and see where it goes, hence Micro Chat, but the other part is to gauge the sentiment in the room and see whether we can actually fund something much bigger totally aligned with the community. I'm a huge fan of Google services and the utility they offer, but the nature of technology has veered so far into addiction with scrolling feeds, streaming and social media that I feel there's a need to do something now.
If there's interest, please comment and provide feedback.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.Project mention: A Deep Dive into the Design of Directory Quotas in JuiceFS | dev.to | 2023-10-27
If you have any questions or would like to learn more, feel free to join discussions about JuiceFS on GitHub and the JuiceFS community on Slack.
Temporal serviceProject mention: temporal VS laravel-workflow - a user suggested alternative | libhunt.com/r/temporal | 2023-08-23
Distributed, offline-first bug tracker embedded in git, with bridges
Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily. Try it. Test it. If you feel it's better, use it! 𝐉𝐚𝐯𝐚有𝐝𝐮𝐛𝐛𝐨, 𝐆𝐨𝐥𝐚𝐧𝐠有𝐫𝐩𝐜𝐱! build for cloud!
Golang implementation of the Raft consensus protocolProject mention: Leader election library | /r/golang | 2023-07-07
Depending on your exact needs, you could try HashiCorp's Raft implementation: https://github.com/hashicorp/raft
Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.Project mention: Ask HN: Who is hiring? (December 2023) | news.ycombinator.com | 2023-12-01
Uber | Software Engineers | Hybrid (Denmark) | https://www.uber.com/dk/en/careers/locations/aarhus/
Work with an amazing team responsible for the infrastructure software that makes Uber’s data centers around the world reliable and scalable. If you want to solve the toughest engineering challenges alongside some of the smartest people in the industry, Uber Aarhus is the right place for you.
The team in Aarhus build and operate the stateless and stateful compute platforms used by nearly every other engineer in the company (Up - https://www.uber.com/en-GB/blog/up-portable-microservices-re... and Odin - https://www.uber.com/en-GB/blog/how-uber-optimized-cassandra...) as well as other related infrastructure projects such as Cadence - https://github.com/uber/cadence.
Go RPC framework with high-performance and strong-extensibility for building micro-services.
Data-Centric Pipelines and Data VersioningProject mention: Show HN: We scaled Git to support 1 TB repos | news.ycombinator.com | 2022-12-13
There are a couple of other contenders in this space. DVC (https://dvc.org/) seems most similar.
If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.)
The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake.
Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
Ultra performant API Gateway with middlewares. A project hosted at The Linux Foundation
A Cloud Native traffic orchestration systemProject mention: Thoughts on new tool - easegress - 5k stars in 8 months | /r/kubernetes | 2023-03-03
Since there is no discussion on reddit, wanted to get people's thoughts on a relatively new project easegress which looks to have gained a lot of stars. Seems interesting and something that can't easily be compared to any other individual tool since it combines a lot of features in one.
⟁ Tendermint Core (BFT Consensus) in GoProject mention: There aren't that many uses for blockchains | /r/programming | 2023-03-16
There are good use-cases, but to much focus on cryptocurrencies. Frameworks are not well designed to be used in other use-cases. Best that I have found that doesn't forces you to a specific architecture is Tendermint, but is not even close to state-of-the-art performance and scalability. Everything else is designed around cryptocurrency and smart contracts.
Build powerful pipelines in any programming language.
Full-featured BitTorrent client package and utilitiesProject mention: 0x0: Share Files from Terminal | news.ycombinator.com | 2023-04-08
https://github.com/anacrolix/torrent/blob/master/cmd/torrent... does exactly that. Install with `go get github.com/anacrolix/torrent/cmd/torrent@latest`, and then run `torrent serve `.
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Go Distributed Systems related posts
Adding new database engine support
1 project | /r/stalwartlabs | 25 Nov 2023
Transitioning from more traditional OOP like C# to Go, what are the biggest coding style differences.
2 projects | /r/golang | 16 Nov 2023
Build Your Own BitTorrent
9 projects | news.ycombinator.com | 19 Oct 2023
Spacedrive: Unify files from all your devices and clouds into one easy explorer
2 projects | news.ycombinator.com | 19 Oct 2023
Durable Coroutines for Go
2 projects | news.ycombinator.com | 18 Oct 2023
Fault Tolerance in Distributed Systems: Strategies and Case Studies
4 projects | dev.to | 18 Oct 2023
Coroutines for Golang
1 project | news.ycombinator.com | 17 Oct 2023
A note from our sponsor - #<SponsorshipServiceOld:0x00007f0fa33642f0>
www.saashub.com | 3 Dec 2023
What are some of the best open-source Distributed System projects in Go? This list will help you: