Top 23 Python Distributed Projects

Distributed

• Add a project

1. Ray

   1 47 36,132 10.0 Python

   Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

   

   Project mention: Ask HN: What Open Source Projects Need Help? | news.ycombinator.com | 2024-11-16

   I'm guessing this comment is some kind of "if you know, you know." Likely starting from https://docs.ray.io/en/latest/cluster/vms/user-guides/launch... and then trawling through one of these I guess https://github.com/ray-project/ray/issues?q=is%3Aissue+prem+...

2. CodeRabbit

   coderabbit.ai featured

   CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

   

3. optuna

   2 34 11,583 9.9 Python

   A hyperparameter optimization framework

   

   Project mention: Optuna – A Hyperparameter Optimization Framework | news.ycombinator.com | 2024-04-06

   I didn’t even know WandB did hyperparameter optimization, I figured it was a neural network visualizer based on 2 minute papers. Didn’t seem like many alternatives out there to Optuna with TPE + persistence in conditional continuous & discrete spaces.

   Anyway, it’s doable to make a multi objective decide_to_prune function with Optuna, here’s an example https://github.com/optuna/optuna/issues/3450#issuecomment-19...

4. modin

   3 11 10,063 8.5 Python

   Modin: Scale your Pandas workflows by changing a single line of code

   

5. scrapy-redis

   4 4 5,583 6.0 Python

   Redis-based components for Scrapy.

   

6. hatchet

   5 18 4,637 9.8 Python

   A distributed, fault-tolerant task queue

   

   Project mention: Running Durable Workflows in Postgres Using DBOS | news.ycombinator.com | 2024-12-10

   Disclaimer: I'm a co-founder of Hatchet (https://github.com/hatchet-dev/hatchet), which is a Postgres-backed task queue that supports durable execution.

   > Because a step transition is just a Postgres write (~1ms) versus an async dispatch from an external orchestrator (~100ms), it means DBOS is 25x faster than AWS Step Functions

   Durable execution engines deployed as an external orchestrator will always been slower than direct DB writes, but the 1ms delay versus ~100ms doesn't seem inherent to the orchestrator being external. In the case of Hatchet, pushing work takes ~15ms and invoking the work takes ~1ms if deployed in the same VPC, and 90% of that execution time is on the database. In the best-case, the external orchestrator should take 2x as long to write a step transition (round-trip network call to the orchestrator + database write), so an ideal external orchestrator would be ~2ms of latency here.

   There are also some tradeoffs to a library-only mode that aren't discussed. How would work that requires global coordination between workers behave in this model? Let's say, for example, a global rate limit -- you'd ideally want to avoid contention on rate limit rows, assuming they're stored in Postgres, but each worker attempting to acquire a rate limit simultaneously would slow down start time significantly (and place additional load on the DB). Whereas with a single external orchestrator (or leader election), you can significantly increase throughput by acquiring rate limits as part of a push-based assignment process.

   The same problem of coordination arises if many workers are competing for the same work -- for example if a machine crashes while doing work, as described in the article. I'm assuming there's some kind of polling happening which uses FOR UPDATE SKIP LOCKED, which concerns me as you start to scale up the number of workers.

7. Gerapy

   6 1 3,414 5.4 Python

   Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

   

8. lingvo

   7 1 2,833 8.2 Python

   Lingvo

   

9. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

10. arq

    8 4 2,390 6.2 Python

    Fast job queuing and RPC in python with asyncio and redis.

    

11. fugue

    9 11 2,055 5.7 Python

    A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

    

12. trainer

    10 2 1,721 9.2 Python

    Distributed ML Training and Fine-Tuning on Kubernetes

    

13. MLBox

    11 1 1,503 0.0 Python

    MLBox is a powerful Automated Machine Learning python library.

    

14. quokka

    12 23 1,157 2.9 Python

    Making data lake work for time series (by marsupialtail)

    

15. pottery

    13 5 1,133 7.3 Python

    Redis for humans. 🌎🌍🌏

    

16. code2vec

    14 3 1,126 2.5 Python

    TensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"

    

17. evotorch

    15 14 1,044 4.9 Python

    Advanced evolutionary computation library built directly on top of PyTorch, created at NNAISENSE.

    

18. runhouse

    16 7 1,014 9.9 Python

    Distribute and run AI workloads magically in Python, like PyTorch for ML infra.

    

    Project mention: Show HN: RAG app example with self-hosted embedding and LLM services | news.ycombinator.com | 2024-08-13

19. bagua

    17 6 879 1.9 Python

    Bagua Speeds up PyTorch

    

20. modal-examples

    18 9 807 9.6 Python

    Examples of programs built using Modal

    

21. optuna-examples

    19 2 729 8.5 Python

    Examples for https://github.com/optuna/optuna

    

22. AgileRL

    20 12 701 9.3 Python

    Streamlining reinforcement learning with RLOps. State-of-the-art RL algorithms and tools.

    

23. Pyrlang

    21 4 613 1.4 Python

    Erlang node implemented in Python 3.5+ (Asyncio-based)

    

24. wakaq

    22 3 581 8.0 Python

    Background task queue for Python backed by Redis, a super minimal Celery

    

25. malib

    23 2 518 3.4 Python

    A parallel framework for population-based multi-agent reinforcement learning.

    

26. SaaSHub

    www.saashub.com featured

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    

Python Distributed related posts

• Ask HN: Going beyond Pandas for analysis, how to stay sane?

  1 project | news.ycombinator.com | 14 Feb 2025

• Show HN: Dataclr – Python library simplifying feature selection for ML

  1 project | news.ycombinator.com | 6 Jan 2025

• Dataclr – New feature selection algorithm for ML achieving SOTA results

  1 project | news.ycombinator.com | 5 Jan 2025

• Github's Top 31 items of Dec 18, 2024

  1 project | dev.to | 18 Dec 2024

• AIM Weekly 28 Oct 2024

  21 projects | dev.to | 28 Oct 2024

• Multimodal Madness! Create a Product Recommender for Smart Shopping

  6 projects | dev.to | 7 Aug 2024

• Comparison: Dask vs. Ray

  1 project | news.ycombinator.com | 14 Jun 2024
• A note from our sponsor - CodeRabbit
  coderabbit.ai | 22 Mar 2025

  Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR. Learn more →

Index

Sponsored

CodeRabbit: AI Code Reviews for Developers

Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

coderabbit.ai

Do not miss the trending Python projects with our weekly report!