Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Distributed System Open-Source Projects
-
advanced-java
😮 Core Interview Questions & Answers For Experienced Java(Backend) Developers | 互联网 Java 工程师进阶知识完全扫盲：涵盖高并发、分布式、高可用、微服务、海量数据处理等领域知识
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
spacedrive
Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.
-
xgboost
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
seaweedfs
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC active-active replication, Kubernetes, POSIX FUSE mount, S3 API, S3 Gateway, Hadoop, WebDAV, encryption, Erasure Coding.
-
-
-
-
Nomad
Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
-
-
-
-
-
NebulaGraph Database
A distributed, fast open-source graph database featuring horizontal scalability and high availability (by vesoft-inc)
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Oracle Linux 8.8'de PostgreSQL 13 Yedekli Yapı Nasıl Kurulur? - Patroni, ETCD, HAProxy | dev.to | 2023-12-07sudo dnf -y install curl wget vim ETCD_RELEASE=$(curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest|grep tag_name | cut -d '"' -f 4) echo $ETCD_RELEASE wget https://github.com/etcd-io/etcd/releases/download/${ETCD_RELEASE}/etcd-${ETCD_RELEASE}-linux-amd64.tar.gz tar xvf etcd-${ETCD_RELEASE}-linux-amd64.tar.gz cd etcd-${ETCD_RELEASE}-linux-amd64 sudo mv etcd* /usr/local/bin ls /usr/local/bin /usr/local/bin/etcd --version
In the video, Mo talked about a few packages like Cidre and StrOm, and we referred to SpaceDrive.
Project mention: Learn how to design systems at scale and prepare for system design interviews | news.ycombinator.com | 2023-07-05There are things in that doc that are completly wrong and ridiculous such as:
https://github.com/karanpratapsingh/system-design#streaming-...
Project mention: NSQ: Open-source realtime distributed messaging, billions of messages / day | news.ycombinator.com | 2024-01-09
Project mention: DwarFS – The Deduplicating Warp-Speed Advanced Read-Only File System | news.ycombinator.com | 2024-04-11Whoops: WebDAV:
https://news.ycombinator.com/item?id=39417503
SeaweedFS supports WebDAV. https://github.com/seaweedfs/seaweedfs/wiki/WebDAV
I'm not able to find if both/restic supports mounting backups as WebDAV, but in theory there's nothing stopping you.
It's 100% user space (expose a rest service) and supported by a bunch of file-browsers with a bit of a network aware component to it as well.
Project mention: Reverse Engineering Protobuf Definitions from Compiled Binaries | news.ycombinator.com | 2024-03-09The reflection service is open-sourced (at least for some sdks):
* https://github.com/grpc/grpc-go/blob/master/Documentation/se...
* https://chromium.googlesource.com/external/github.com/grpc/g...
Project mention: The lightweight, easy-to-use, distributed relational database built on SQLite | news.ycombinator.com | 2024-02-23
Project mention: Implementing OTel Trace Context Propagation Through Message Brokers with Go | dev.to | 2024-04-04Several message brokers, such as NATS and database queues, are not supported by OpenTelemetry (OTel) SDKs. This article will guide you on how to use context propagation explicitly with these message queues.
I don't have any further insight, but looking at <https://github.com/hashicorp/nomad/forks?include=active&page...> coughed up https://github.com/atlassian/nomad/branches although confusingly it says "updated last week" but browsing any one of the branches seems to be stupid old so I got nothing
Finding conceptual forks, e.g. $(git push --mirror ...) would be trickier but I bet sourcegraph could do it
Ultimately, the question boils down to: what risk are you driving down: hitching your wagon to a dead stack, not getting security updates, not getting PRs merged, $other?
Kotlin also has a construct for asynchronous collections/streams. Kotlin's version of AsyncSequence is called a Flow. Just as Swift's AsyncSequence builds upon prior experience with RxSwift and Combine, Kotlin's Flow APIs build upon earlier stream/collection APIs in the JVM ecosystem: Java's RxJava, Java8 Streams, Project Reactor, and Scala's Akka.
Apache ZooKeeper — a distributed coordination, synchronization, and configuration service (written in Java);
Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08
Project mention: Ask HN: How do I find technical papers (for distributed systems)? | news.ycombinator.com | 2023-07-03Distributed systems is a very broad research topic nowadays, so it might make sense to check conferences covering a narrower topic, e.g. Cloud computing. Google Scholar might give some useful results for more specific search terms.
The reading lists for distributed systems university courses might also be interesting as well as this "awesome" link list on github:
https://github.com/theanalyst/awesome-distributed-systems
We should chat more about Reminders v2 - I want to learn more about what you'd like it to provide. Can you comment on the issue (https://github.com/dotnet/orleans/issues/7573)? If you'd like to see some of what is in the works, please message me on Discord and we can chat more about it (https://aka.ms/orleans/discord)
I don't know if I agree with the argument regarding durability vs elastic execution. If I can get both (with a nice API/DX) via something like Temporal (https://github.com/temporalio/temporal), what's the drawback here?
Project mention: Need insights to build a distributed key value store from scratch. | /r/DistributedComputing | 2023-12-08Pls check this course: https://github.com/pingcap/talent-plan . It includes how to implement sql and key value store. It’s an awesome course
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Distributed Systems related posts
- Micro – A Go service development platform
- IBM Planning to Acquire HashiCorp
- SwiftPaxos: Fast Geo-Replicated State Machines
- How to Visualize and Analyze Data in Open Source Communities
- SwiftPaxos: Fast Geo-Replicated State Machines
- SwiftPaxos: Fast Geo-Replicated State Machines
- Interview with Mo Rajabi, co-founder and CEO of Noor
-
A note from our sponsor - InfluxDB
www.influxdata.com | 26 Apr 2024
Index
What are some of the best open-source Distributed System projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | advanced-java | 74,203 |
| 2 | awesome-scalability | 53,036 |
| 3 | etcd | 46,345 |
| 4 | Dubbo | 40,021 |
| 5 | spacedrive | 28,841 |
| 6 | system-design | 28,737 |
| 7 | xgboost | 25,576 |
| 8 | nsq | 24,561 |
| 9 | go-micro | 21,393 |
| 10 | seaweedfs | 21,013 |
| 11 | grpc-go | 19,870 |
| 12 | rqlite | 14,862 |
| 13 | NATS | 14,720 |
| 14 | Nomad | 14,422 |
| 15 | Akka | 12,925 |
| 16 | micro | 12,017 |
| 17 | Apache ZooKeeper | 11,925 |
| 18 | conductor | 10,230 |
| 19 | awesome-distributed-systems | 10,165 |
| 20 | NebulaGraph Database | 10,114 |
| 21 | Orleans | 9,821 |
| 22 | temporal | 9,806 |
| 23 | talent-plan | 9,800 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com