Python Distributed Systems

Open-source Python projects categorized as Distributed Systems
Edit details
Topics: Python Machine Learning Asyncio Deep Learning Distributed Computing
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 23 Python Distributed System Projects

Distributed Systems

  • petals

    🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

    • Project mention: Chameleon: Meta's New Multi-Modal LLM | news.ycombinator.com | 2024-05-21

    Things like [petals](https://github.com/bigscience-workshop/petals) exist, distributed computing over willing participants. Right now corporate cash is being rammed into the space so why not snap it up while you can, but the moment it dries up projects like petals will see more of the love they deserve.

    I envision a future where crypto-style booms happen over tokens useful for purchasing priority computational time, which is earned by providing said computational time. This way researchers can daisy-chain their independent smaller rigs together into something with gargantuan capabilities.

  • Scout Monitoring

    Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

    Scout Monitoring logo

  • Faust

    Python Stream Processing

    • Project mention: Faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust | 2023-12-07

  • faststream

    FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

    • Project mention: FastStream v0.4.0: Introducing Confluent Kafka Integration with Async Support | news.ycombinator.com | 2024-01-30

  • fugue

    A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

    • Project mention: FLaNK Stack Weekly 22 January 2024 | dev.to | 2024-01-22

  • hivemind

    Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

    • Project mention: You can now train a 70B language model at home | news.ycombinator.com | 2024-03-07

    https://github.com/learning-at-home/hivemind is also relevant

  • system-design-questions

    Problem statements on System Design and Software Architecture as part of Arpit's System Design Masterclass

  • faust

    Python Stream Processing. A Faust fork (by faust-streaming)

    • Project mention: faust VS quix-streams - a user suggested alternative | libhunt.com/r/faust-streaming/faust | 2023-12-07

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • eventsourcing

    A library for event sourcing in Python.

  • vizier

    Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

  • xorbits

    Scalable Python DS & ML, in an API compatible & lightning fast way.

  • bagua

    Bagua Speeds up PyTorch

  • rd-blender-docker

    A collection of Docker containers for running Blender headless or distributed ✨

  • adaptdl

    Resource-adaptive cluster scheduler for deep learning training.

  • chat.petals.dev

    💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client

    • Project mention: Make no mistake—AI is owned by Big Tech | /r/transhumanism | 2023-12-07

    ETA: https://chat.petals.dev

  • Anubis

    Distributed LMS for automating Computing Science Courses From NYU

  • tractor

    A distributed, structured concurrent runtime for Python (and friends)

    • Project mention: Ask HN: What Python libraries do you wish more people knew about? | news.ycombinator.com | 2023-12-03

  • Thespian Actor Library

    Python Actor concurrency library

  • checkmate

    Training neural networks in TensorFlow 2.0 with 5x less memory (by parasj)

  • zef

    Toolkit for graph-relational data across space and time (by zefhub)

  • zimfarm

    Farm operated by bots to grow and harvest new zim files

  • walnats

    🌰 Nats-powered event-driven background jobs and microservices framework for Python.

  • hyx

    🧘‍♀️ Lightweight fault tolerant primitives for your modern asyncio Python microservices

  • easyrpc

    An easy to use rpc framework for enabling fast inter-process, inter-container, or inter-host communication

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Distributed Systems discussion

Log in or Post with

Python Distributed Systems related posts

Index

What are some of the best open-source Distributed System projects in Python? This list will help you:

Project Stars
1 petals 8,819
2 Faust 6,688
3 faststream 1,943
4 fugue 1,902
5 hivemind 1,856
6 system-design-questions 1,747
7 faust 1,495
8 eventsourcing 1,415
9 vizier 1,184
10 xorbits 1,034
11 bagua 868
12 rd-blender-docker 409
13 adaptdl 407
14 chat.petals.dev 299
15 Anubis 287
16 tractor 258
17 Thespian Actor Library 185
18 checkmate 125
19 zef 108
20 zimfarm 81
21 walnats 64
22 hyx 64
23 easyrpc 63

Sponsored
Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com