Python Distributed Systems

Open-source Python projects categorized as Distributed Systems
Edit details
Python Machine Learning Asyncio Distributed Computing Deep Learning
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 23 Python Distributed System Projects

Distributed Systems

  1. petals

    🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

    Project mention: Serving AI from the Basement – 192GB of VRAM Setup | news.ycombinator.com | 2024-09-08

  2. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  3. Faust

    Python Stream Processing

  4. faststream

    FastStream is a powerful and easy-to-use Python framework for building asynchronous services interacting with event streams such as Apache Kafka, RabbitMQ, NATS and Redis.

    Project mention: FastStream: A powerful library for building services with event streams | news.ycombinator.com | 2024-10-29

    FastStream (https://github.com/airtai/faststream) simplifies the process of writing producers and consumers for message queues, handling all the parsing, networking and documentation generation automatically. It is a new package based on the ideas and experiences gained from FastKafka and Propan. By joining our forces, we picked up the best from both packages and created a unified way to write services capable of processing streamed data regardless of the underlying protocol. We'll continue to maintain both packages, but new development will be in this project.

    Making streaming microservices has never been easier. Designed with junior developers in mind, FastStream simplifies your work while keeping the door open for more advanced use cases. Here's a look at the core features that make FastStream a go-to framework for modern, data-centric microservices.

    Multiple Brokers: FastStream provides a unified API to work across multiple message brokers (Apache Kafka, RabbitMQ, NATS and Redis)

  5. PySR

    High-Performance Symbolic Regression in Python and Julia

    Project mention: Genetically synthesized supergain broadband wire-bundle antenna | news.ycombinator.com | 2024-07-31

    For those who're only distantly aware of the kind of problem this solves (like me), the wikipedia link further elaborates:

    https://en.wikipedia.org/wiki/Symbolic_regression

    and turns out there's a Python package

    https://github.com/MilesCranmer/PySR

    I've needed something like this at least once (but IIRC no more than twice ;) ), so I'm glad to know what to look for next time, thanks for the rabbit hole!

  6. system-design-questions

    Problem statements on System Design and Software Architecture as part of Arpit's System Design Masterclass

    Project mention: 🔥 17 Best Free GitHub Repositories to Crack System Design Interviews 🛠️ | dev.to | 2024-12-06

    13. System Design questions by Arpit Bhayani

  7. hivemind

    Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

    Project mention: You can now train a 70B language model at home | news.ycombinator.com | 2024-03-07

    https://github.com/learning-at-home/hivemind is also relevant

  8. fugue

    A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

  9. faust

    Python Stream Processing. A Faust fork (by faust-streaming)

  10. vizier

    Python-based research interface for blackbox and hyperparameter optimization, based on the internal Google Vizier Service.

  11. eventsourcing

    A library for event sourcing in Python.

  12. xorbits

    Scalable Python DS & ML, in an API compatible & lightning fast way.

  13. bagua

    Bagua Speeds up PyTorch

  14. rd-blender-docker

    A collection of Docker containers for running Blender headless or distributed ✨

  15. adaptdl

    Resource-adaptive cluster scheduler for deep learning training.

  16. open-box

    Generalized and Efficient Blackbox Optimization System (by PKU-DAIR)

  17. chat.petals.dev

    💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client

  18. Anubis

    Distributed LMS for automating Computing Science Courses From NYU

  19. tractor

    A distributed, structured concurrent runtime for Python (and friends)

  20. Thespian Actor Library

    Python Actor concurrency library

  21. checkmate

    Training neural networks in TensorFlow 2.0 with 5x less memory (by parasj)

  22. zef

    Toolkit for graph-relational data across space and time (by zefhub)

  23. zimfarm

    Farm operated by bots to grow and harvest new zim files

  24. walnats

    🌰 Nats-powered event-driven background jobs and microservices framework for Python.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Distributed Systems discussion

Log in or Post with

Python Distributed Systems related posts

Index

What are some of the best open-source Distributed System projects in Python? This list will help you:

# Project Stars
1 petals 9,366
2 Faust 6,758
3 faststream 3,348
4 PySR 2,569
5 system-design-questions 2,125
6 hivemind 2,099
7 fugue 2,035
8 faust 1,699
9 vizier 1,531
10 eventsourcing 1,492
11 xorbits 1,140
12 bagua 877
13 rd-blender-docker 436
14 adaptdl 428
15 open-box 395
16 chat.petals.dev 308
17 Anubis 302
18 tractor 273
19 Thespian Actor Library 190
20 checkmate 128
21 zef 113
22 zimfarm 88
23 walnats 81

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
Loading...