Top 21 Python Distributed Projects
-
Ray
An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.
Project mention: Is it normal to have a negative and near-zero explained variance in PPO? | reddit.com/r/reinforcementlearning | 2021-12-25I guess I did, as I directly use the PPO agent provided by the RLlib.
-
nni
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
Project mention: Automated Machine Learning (AutoML) - 9 Different Ways with Microsoft AI | dev.to | 2021-10-04For a complete tutorial, navigate to this Jupyter Notebook: https://github.com/microsoft/nni/blob/master/examples/notebooks/tabular_data_classification_in_AML.ipynb
-
SonarLint
Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.
-
Project mention: TIL about modin.pandas which significantly speeds up pandas if you import modin.pandas instead of pandas. | reddit.com/r/u_pygsm | 2021-06-30
Source
-
Project mention: Trading Algos - 5 Key Metrics and How to Implement Them in Python | dev.to | 2022-01-08
Nothing can beat iteration and rapid optimization. Try running things like grid experiments, batch optimizations, and parameter searches. Take a look at various packages like hyperopt or optuna as packages that might be able to help you here!
-
There are many different Scrapyd dashboard and admin tools available, from ScrapeOps (Live Demo) to SpiderKeeper, and Gerapy.
-
-
Project mention: I made a simple async queueing framework called SAQ! It includes a built in web UI to manage jobs. | reddit.com/r/Python | 2022-01-06
I need to process a lot of long running IO heavy jobs with background workers. I've been using ARQ for a while but decided to take a crack at writing my own distributed queue.
-
OPS
OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.
-
code2vec
TensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"
Project mention: [D] Security feature labeled dataset for code2vec | reddit.com/r/MachineLearning | 2021-10-09I am looking for a dataset that would contain code snippets (or vector representing it) and labels that are security specific features such as authentication, encryption, logging etc. I need to apply techniques like code2vec https://github.com/tech-srl/code2vec but with security-specific labels. Any leads where can I find this kind of dataset?
-
Project mention: Bagua: Speed up and Scale PyTorch (r/MachineLearning) | reddit.com/r/datascienceproject | 2021-10-16
-
Project mention: Worth wrapping pottery functions for compliance with async? | reddit.com/r/Python | 2021-08-01
I have a question about https://github.com/brainix/pottery. It provides a nice Pythonic API by wrapping Redis constructs with Python Redis-backed data structures (Dict, Deque, etc.). I am using it in a Fastapi microservice project, which is obviously async.
-
- Python: https://github.com/Pyrlang/Pyrlang
-
fugue
A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark and Dask without any rewrites.
There's dask-sql, but I think it is being abandoned for fugue-project. I'm actually excited for this project as it is trying to provide a backend agnostic solution, which would seem like a difficult, lofty goal. I wish them luck.
-
PySR
Simple, fast, and parallelized symbolic regression in Python/Julia via regularized evolution and simulated annealing
Project mention: [D] Inferring general physical laws from observations in 300 lines of code | reddit.com/r/MachineLearning | 2021-08-02This is really neat! Since you're interested in this subject, you may also appreciate PySR and the corresponding paper which uses Graph Neural Networks to perform symbolic regression.
-
machin
Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...
Project mention: Best PyTorch RL library for doing research | reddit.com/r/reinforcementlearning | 2021-04-30Machin is really nice, it is very easy to use and to try different things, although it’s developed by one person and maybe not appropriately tested yet.
-
Project mention: MALib: A parallel framework for population-based multi-agent reinforcement learning | reddit.com/r/reinforcementlearning | 2021-07-23
Code for https://arxiv.org/abs/2106.07551 found: https://github.com/sjtu-marl/malib
-
lithops
An open source framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud.
Project mention: [D] For those of you who don't own a GPU, how do you run your experiments or train your models? | reddit.com/r/MachineLearning | 2021-12-19At work for non-ML/non-GPU stuff I've been using Lithops for running code on dynamically-provisioned cloud resources (serverless or VM). It pickles your code & runtime variables, sends them to cloud storage, runs the code & downloads the results, all relatively transparently. You're just calling Python functions with Python objects on your local computer and not having to worry about deploying your code, packaging your data, etc. Better still, you can scale up for things like hyperparameter sweeps by just dispatching more calls in parallel, and it will provision more resources.
-
Ignareo-ISML-auto-voter
Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns!
Project mention: Web crawler/spider of ultimate concurrency packed as microservice nodes | news.ycombinator.com | 2021-10-22 -
That's still regular ML because you are in charge of the features. Optuna might make your life easier though: https://github.com/optuna/optuna-examples/blob/main/xgboost/xgboost_simple.py
-
More code samples here: https://github.com/hazelcast/hazelcast-python-client/tree/master/examples
-
GitHub of the VPN software itself: https://github.com/LetheanMovement/lethean-vpn
-
Project mention: Show HN: Hookdeck, an Infrastructure to Consume Webhooks | news.ycombinator.com | 2021-08-04
Python Distributed related posts
- Discussion: the feasubility of using open source hyperparameter optimization tools and SQLAlchemy to automatically tune database performance
- I made a simple async queueing framework called SAQ! It includes a built in web UI to manage jobs.
- Is it normal to have a negative and near-zero explained variance in PPO?
- [PPO] Reward graph fluctuates a lot after certain timesteps
- What are good alternatives to zip files when working with large online image datasets?
- JORLDY: OpenSource Reinforcement Learning Framework
- [P] optimization of Hugging Face Transformer models to get Inference < 1 Millisecond Latency + deployment on production ready inference server
Index
What are some of the best open-source Distributed projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Ray | 18,887 |
| 2 | nni | 10,884 |
| 3 | modin | 6,704 |
| 4 | optuna | 5,808 |
| 5 | Gerapy | 2,635 |
| 6 | lingvo | 2,371 |
| 7 | arq | 1,072 |
| 8 | code2vec | 825 |
| 9 | bagua | 634 |
| 10 | pottery | 610 |
| 11 | Pyrlang | 489 |
| 12 | fugue | 475 |
| 13 | PySR | 467 |
| 14 | machin | 262 |
| 15 | malib | 250 |
| 16 | lithops | 181 |
| 17 | Ignareo-ISML-auto-voter | 165 |
| 18 | optuna-examples | 134 |
| 19 | hazelcast-python-client | 97 |
| 20 | lethean-vpn | 31 |
| 21 | python-ksuid | 31 |
Are you hiring? Post a new remote job listing for free.
