Top 3 Python Parallelization Projects

swifter

3 2,459 5.5 Python

A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner (by jmcarpenter2)
covalent

4 689 8.9 Python

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)

Project mention: Remote execution of code | /r/Python | 2023-12-05

Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
magda

1 13 4.1 Python

Library for building Modular and Asynchronous Graphs with Directed and Acyclic edges (MAGDA)

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).