The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 23 Scalability Open-Source Projects
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
TDengine
TDengine is an open source, high-performance, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, Industrial IoT and DevOps.
-
PaddlePaddle
PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
NebulaGraph Database
A distributed, fast open-source graph database featuring horizontal scalability and high availability (by vesoft-inc)
-
Centrifugo
Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably. Set up once and forever.
-
Hazelcast
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
-
-
-
System-Design
It's just fascinating. How is modern software designed? 🤔 Some design-level considerations for scalability, maintainability eventual consistency, availability & reliability. 👨💻 Interview Prep. 👨💻 (by codersguild)
-
MooseFS
MooseFS – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System (Software-Defined Storage)
-
shadow
Shadow is a discrete-event network simulator that directly executes real application code, enabling you to simulate distributed systems with thousands of network-connected processes in realistic and scalable private network experiments using your laptop, desktop, or server running Linux. (by shadow)
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: A collection of learning resources for curious software engineers | news.ycombinator.com | 2024-02-19The inclusion of the perspective section: https://github.com/charlax/professional-programming?tab=read... I think is really smart. Same for personal productivity. Two things that can dramatically change how and what you end up studying and doing with your time / life.
I did a coding bootcamp and yeah the frontend knowledge they taught was useful, but I could have learned that online for free. Looking back, the far more valuable thing I learned was how to discipline myself and my time - that was the first time in my life I was truly disciplined and mindful in how I spent my time. I also got perspective I'd never seen before: there was some folks in my cohort that were in their 30s and 40s and undergoing career change, and I learned two things from them: First, don't stress too much, your life has much more flexibility than you might expect (this truth is borne out, they all have perfectly successful careers in their new lives as engineers), and second, make a great use of the time you have.
Bog-standard advice we all know, but to witness it firsthand from people living it and sharing it is different. The shared article in the github is incredible: https://www.theguardian.com/commentisfree/2020/sep/07/termin...
I often wonder why I don't see more of these sorts of articles. From watching a family member slowly die of cancer, and from reading books like "When Breath Becomes Air," I'm guessing it's some combination of exhaustion, disability, and a new set of priorities that doesn't really involve death blogging. Still, I find these kinds of writings more poignant than most things I read.
Project mention: Learn how to design systems at scale and prepare for system design interviews | news.ycombinator.com | 2023-07-05There are things in that doc that are completly wrong and ridiculous such as:
https://github.com/karanpratapsingh/system-design#streaming-...
Click to Learn more...
Project mention: 24 GitHub repos with 372M views that you can't miss out as a software engineer | dev.to | 2024-01-25A curated list of Site Reliability and Production Engineering resources: https://github.com/dastergon/awesome-sre
VictoriaMetrics CTO here.
The referred library is the official OpenTelemetry package for reading metrics in Go language [1] - more details are available at [2].
Note that we at VictoriaMetrics like the idea of unified observability standard like OpenTelemetry. The issue is in the current otel implementation. It is too bloated and very inefficient. This contradicts to our experience with observability cases, which need very optimized format for metrics' transfer in order to reduce costs on CPU and network traffic needed to transfer and process these metrics.
VictoriaMetrics continues investing in OpenTelemetry by providing integration docs [3] and improving the existing functionality for otel metrics' ingestion [4].
[1] https://github.com/open-telemetry/opentelemetry-proto-go
[2] https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2570...
[3] https://docs.victoriametrics.com/guides/getting-started-with...
[4] https://github.com/VictoriaMetrics/VictoriaMetrics/issues/60...
Project mention: WebSockets vs. Server-Sent-Events vs. Long-Polling vs. WebRTC vs. WebTransport | news.ycombinator.com | 2024-03-20Hello, I am author of https://github.com/centrifugal/centrifugo. Our users can choose from WebSocket, EventSource, WebTransport (experimental stabilize in the future). WebRTC is out of scope as the main purpose is central server based real-time json/binary messaging, and WebRTC makes things much more complex since it shines for peer-to-peer and rich media communications.
What I'd like to add is that Centrifugo also supports HTTP-streaming – not mentioned by the OP – but this is a transport which has advantages over Eventsource - like possibility to send POST body on initial request from web browser (with SSE you can not), it supports binary, and with Readable Streams browser API it's widely supported by modern browsers.
Another thing I'd like to mention about Centrifugo - it supports bidirectional WebSocket fallbacks with EventSource and HTTP-streaming, and does this without sticky sessions requirement. I guess nobody else have this at this point. See https://centrifugal.dev/blog/2022/07/19/centrifugo-v4-releas.... Which solves one more practical concern. Sticky sessions is an optimization in Centrifugo case, not a requirement.
If you are interested in topic, we also have a post about WebSocket scalability - https://centrifugal.dev/blog/2020/11/12/scaling-websocket - it covers some design decisions made in Centrifugo.
Project mention: Does anyone know any good java implementations for distributed key-value store? | /r/ExperiencedDevs | 2023-06-08You're probably looking for Hazelcast here. Note that it does much more than just a distributed k/v, but it will get you where you need to go.
https://github.com/Xudong-Huang/may
The project has some serious restrictions and unsound footguns (e.g. around TLS), but otherwise it's usable enough. There are also a number of C/C++ libraries, but I can not comment on those.
Project mention: Turmoil, a framework for developing and testing distributed systems | news.ycombinator.com | 2023-08-17Cool, will be interested to see how this develops! tokio's loom framework has been a big help in testing some tricky concurrency code I've worked on.
Folks interested in this space might also be interested in the system I spend most of my time working on: Shadow. It also performs deterministic simulation of a network of hosts, but it intercepts network and system interactions at the syscall level via seccomp. As such it can work with binaries compiled from ~any language, usually without any code modification or special compilation. https://shadow.github.io/
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Scalability related posts
- A collection of learning resources for curious software engineers
- Professional Programming – Learning resources for software engineers
- Centrifugo v5 - new major release of real-time messaging server written in Go
- Roadmap and Deadlines
- Centrifuge - multi-transport real-time messaging library with built-in scalability
- Mlstyle.jl: “Functionalprogramming.jl”
- Elixir or golang which wiil be good for large websocket connections.
-
A note from our sponsor - WorkOS
workos.com | 25 Apr 2024
Index
What are some of the best open-source Scalability projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | awesome-scalability | 53,036 |
| 2 | professional-programming | 45,235 |
| 3 | system-design | 28,737 |
| 4 | TDengine | 22,789 |
| 5 | PaddlePaddle | 21,584 |
| 6 | system-design-resources | 14,216 |
| 7 | awesome-sre | 11,484 |
| 8 | VictoriaMetrics | 10,826 |
| 9 | NebulaGraph Database | 10,114 |
| 10 | ml-engineering | 9,719 |
| 11 | system_design | 8,530 |
| 12 | Centrifugo | 7,914 |
| 13 | system-design | 7,736 |
| 14 | Hazelcast | 5,861 |
| 15 | Tuist | 4,196 |
| 16 | node-api-boilerplate | 3,266 |
| 17 | AxonFramework | 3,224 |
| 18 | cote | 2,310 |
| 19 | System-Design | 1,813 |
| 20 | may | 1,708 |
| 21 | MooseFS | 1,583 |
| 22 | Systemizer | 1,351 |
| 23 | shadow | 1,348 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com