Python Distributed

Open-source Python projects categorized as Distributed | Edit details

Top 18 Python Distributed Projects

  • GitHub repo Ray

    An open source framework that provides a simple, universal API for building distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, and Tune, a scalable hyperparameter tuning library.

    Project mention: JORLDY: OpenSource Reinforcement Learning Framework | reddit.com/r/reinforcementlearning | 2021-11-08

    Distributed RL algorithms are provided using ray

  • GitHub repo nni

    An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.

    Project mention: Automated Machine Learning (AutoML) - 9 Different Ways with Microsoft AI | dev.to | 2021-10-04

    For a complete tutorial, navigate to this Jupyter Notebook: https://github.com/microsoft/nni/blob/master/examples/notebooks/tabular_data_classification_in_AML.ipynb

  • Nanos

    Run Linux Software Faster and Safer than Linux with Unikernels.

  • GitHub repo modin

    Modin: Speed up your Pandas workflows by changing a single line of code

    Project mention: TIL about modin.pandas which significantly speeds up pandas if you import modin.pandas instead of pandas. | reddit.com/r/u_pygsm | 2021-06-30

    Source

  • GitHub repo optuna

    A hyperparameter optimization framework

    Project mention: [P] optimization of Hugging Face Transformer models to get Inference < 1 Millisecond Latency + deployment on production ready inference server | reddit.com/r/MachineLearning | 2021-11-05

    There are plenty of different options to do that in OSS, the most well known being optuna (https://github.com/optuna/optuna).

  • GitHub repo Activeloop Hub

    Dataset format for AI. Easily build and manage datasets for machine and deep learning. Stream data real-time & version-control it. https://activeloop.ai (by activeloopai)

    Project mention: TileDB VS Activeloop hub - a user suggested alternative | libhunt.com/r/TileDB | 2021-10-20
  • GitHub repo lingvo

    Lingvo

  • GitHub repo code2vec

    TensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"

    Project mention: [D] Security feature labeled dataset for code2vec | reddit.com/r/MachineLearning | 2021-10-09

    I am looking for a dataset that would contain code snippets (or vector representing it) and labels that are security specific features such as authentication, encryption, logging etc. I need to apply techniques like code2vec https://github.com/tech-srl/code2vec but with security-specific labels. Any leads where can I find this kind of dataset?

  • Scout APM

    Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.

  • GitHub repo bagua

    Bagua Speeds up PyTorch

    Project mention: Bagua: Speed up and Scale PyTorch (r/MachineLearning) | reddit.com/r/datascienceproject | 2021-10-16
  • GitHub repo pottery

    Redis for humans. 🌎🌍🌏

    Project mention: Worth wrapping pottery functions for compliance with async? | reddit.com/r/Python | 2021-08-01

    I have a question about https://github.com/brainix/pottery. It provides a nice Pythonic API by wrapping Redis constructs with Python Redis-backed data structures (Dict, Deque, etc.). I am using it in a Fastapi microservice project, which is obviously async.

  • GitHub repo Pyrlang

    Erlang node implemented in Python 3.5+ (Asyncio-based)

    Project mention: Ask HN: Is Elixir Still Relevant? | news.ycombinator.com | 2021-04-10

    - Python: https://github.com/Pyrlang/Pyrlang

  • GitHub repo PySR

    Simple, fast, and parallelized symbolic regression in Python/Julia via regularized evolution and simulated annealing

    Project mention: [D] Inferring general physical laws from observations in 300 lines of code | reddit.com/r/MachineLearning | 2021-08-02

    This is really neat! Since you're interested in this subject, you may also appreciate PySR and the corresponding paper which uses Graph Neural Networks to perform symbolic regression.

  • GitHub repo fugue

    A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark and Dask without any rewrites. (by fugue-project)

    Project mention: FugueSQL: SQL-ish for pandas, dask, spark | news.ycombinator.com | 2021-10-11

    Hey, I am the author of Fugue.

    Fugue is a higher level abstraction compared to Ray. It provides unified and non-invasive interfaces for people to use Spark, Dask and Pandas. Ray/Modin is also on our roadmap.

    It provides both Python interface (not pandas-like) and Fugue SQL (standard SQL + extra features). Users can choose the one they are most comfortable with as the semantic layer for distributed computing, they are equivalent.

    With Fugue, most of your logic will be in simple Python/SQL that is framework and scale agnostic. From the mindset to the code, Fugue minimizes your dependency on any specific computing frameworks including Fugue itself.

    Please let me know if you want to learn more. our slack is in the README of the fugue repo

    Fugue repo: https://github.com/fugue-project/fugue

  • GitHub repo machin

    Reinforcement learning library(framework) designed for PyTorch, implements DQN, DDPG, A2C, PPO, SAC, MADDPG, A3C, APEX, IMPALA ...

    Project mention: Best PyTorch RL library for doing research | reddit.com/r/reinforcementlearning | 2021-04-30

    Machin is really nice, it is very easy to use and to try different things, although it’s developed by one person and maybe not appropriately tested yet.

  • GitHub repo malib

    A parallel framework for population-based multi-agent reinforcement learning.

    Project mention: MALib: A parallel framework for population-based multi-agent reinforcement learning | reddit.com/r/reinforcementlearning | 2021-07-23

    Code for https://arxiv.org/abs/2106.07551 found: https://github.com/sjtu-marl/malib

  • GitHub repo Ignareo-ISML-auto-voter

    Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns!

    Project mention: Web crawler/spider of ultimate concurrency packed as microservice nodes | news.ycombinator.com | 2021-10-22
  • GitHub repo hazelcast-python-client

    Hazelcast IMDG Python Client

    Project mention: Contribution to Hazelcast | reddit.com/r/Python | 2021-07-05

    More code samples here: https://github.com/hazelcast/hazelcast-python-client/tree/master/examples

  • GitHub repo lethean-vpn

    Lethean Virtual Private Network (VPN)

    Project mention: Lethean - VPN on Monero base | reddit.com/r/Monero | 2021-02-21

    GitHub of the VPN software itself: https://github.com/LetheanMovement/lethean-vpn

  • GitHub repo python-ksuid

    A pure-Python KSUID implementation

    Project mention: Show HN: Hookdeck, an Infrastructure to Consume Webhooks | news.ycombinator.com | 2021-08-04
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2021-11-08.

Python Distributed related posts

Index

What are some of the best open-source Distributed projects in Python? This list will help you:

Project Stars
1 Ray 18,352
2 nni 10,630
3 modin 6,605
4 optuna 5,614
5 Activeloop Hub 3,768
6 lingvo 2,358
7 code2vec 815
8 bagua 598
9 pottery 543
10 Pyrlang 482
11 PySR 450
12 fugue 415
13 machin 247
14 malib 230
15 Ignareo-ISML-auto-voter 165
16 hazelcast-python-client 98
17 lethean-vpn 32
18 python-ksuid 26
Find remote jobs at our new job board 99remotejobs.com. There are 32 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com