Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Python Distributed System Projects
-
petals
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
So how long until we can do an open source Mistral Large?
We could make a start on Petals or some other open source distributed training network cluster possibly?
-
Project mention: Faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust | 2023-12-07
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Hatchet (https://hatchet.run) | New York City | Full-time
We're hiring a founding engineer to help us with development on our open-source, distributed task queue: https://github.com/hatchet-dev/hatchet.
We recently launched on HN, you can check out our launch here: https://news.ycombinator.com/item?id=39643136. We're two second-time YC founders in this for the long haul and we are just wrapping up the YC W24 batch.
As a founding engineer, you'll be responsible for contributing across the entire codebase. We'll compensate accordingly and with high equity. It's currently just the two founders + a part-time contractor. We're all technical and contribute code.
Stack: Typescript/React, Go and PostgreSQL.
To apply, email alexander [at] hatchet [dot] run, and include the following:
1. Tell us about something impressive you've built.
2. Ask a question or write a comment about the state of the project. For example: a file that stood out to you in the codebase, a Github issue or discussion that piqued your interest, a general comment on distributed systems/task queues, or why our code is bad and how you could improve it.
-
fugue
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
-
hivemind
Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
https://github.com/learning-at-home/hivemind is also relevant
-
faststream
FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.
Project mention: FastStream v0.4.0: Introducing Confluent Kafka Integration with Async Support | news.ycombinator.com | 2024-01-30 -
system-design-questions
Problem statements on System Design and Software Architecture as part of Arpit's System Design Masterclass
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Project mention: faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust-streaming/faust | 2023-12-07
-
-
vizier
Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.
-
Project mention: Aviary: Compare Open Source LLMs for cost, latency and quality | news.ycombinator.com | 2023-06-01
-
-
-
-
-
chat.petals.dev
💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client
ETA: https://chat.petals.dev
-
-
Project mention: Ask HN: What Python libraries do you wish more people knew about? | news.ycombinator.com | 2023-12-03
-
-
-
-
-
easyrpc
An easy to use rpc framework for enabling fast inter-process, inter-container, or inter-host communication
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2024-04-01.
Python Distributed Systems related posts
- Mistral Large
- Llama V2 is free to try on the AI Horde
- Correct method to expose "Reference" type in pyo3 (2)
- Aviary: Compare Open Source LLMs for cost, latency and quality
- [N] Aviary: Comparing Open Source LLMs for cost, latency and quality
- Anyscale's Aviary is a dashboard for evaluating Open Source LLMs
- Correct method to expose "Reference" type in pyo3
-
A note from our sponsor - InfluxDB
www.influxdata.com | 18 Apr 2024
Index
What are some of the best open-source Distributed System projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | petals | 8,631 |
| 2 | Faust | 6,671 |
| 3 | hatchet | 2,683 |
| 4 | fugue | 1,869 |
| 5 | hivemind | 1,829 |
| 6 | faststream | 1,758 |
| 7 | system-design-questions | 1,673 |
| 8 | faust | 1,442 |
| 9 | eventsourcing | 1,391 |
| 10 | vizier | 1,171 |
| 11 | ray-llm | 1,126 |
| 12 | xorbits | 997 |
| 13 | bagua | 865 |
| 14 | rd-blender-docker | 409 |
| 15 | adaptdl | 395 |
| 16 | chat.petals.dev | 296 |
| 17 | Anubis | 276 |
| 18 | tractor | 249 |
| 19 | Thespian Actor Library | 185 |
| 20 | checkmate | 123 |
| 21 | zef | 106 |
| 22 | zimfarm | 71 |
| 23 | easyrpc | 60 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com