How to Choose the Right Python Concurrency API

• proxy.py

  5 2,853 1.0 Python

  ⚡ Fast • 🪶 Lightweight • 0️⃣ Dependency • 🔌 Pluggable • 😈 TLS interception • 🔒 DNS-over-HTTPS • 🔥 Poor Man's VPN • ⏪ Reverse & ⏩ Forward • 👮🏿 "Proxy Server" framework • 🌐 "Web Server" framework • ➵ ➶ ➷ ➠ "PubSub" framework • 👷 "Work" acceptor & executor framework

The article gives a good summary of the quite complex landscape of concurrency in python. There's more to it, for example gil-free c-extensions, subprocesses and cross-machine (plus IPC) communication.

But I'm particularly bothered by the fact that many articles and tutorials look at concurrency as if it's only about factoring primes or writing a web server with many (perhaps even idempotent) parallel requests.

In reality, people will often want and need to combine multiple of these approaches, and then it gets VERY messy. I.e. try to combine a multiprocessing executor with multiple asyncio loops and boom you're in some very deep waters.

One project that does this (async loops inside multiple processes) is proxy.py - very enlightening to read its code base [1].

But I really, really wish python would do more to provide simple and robust abstractions for these kinds of tasks. My dream would be a robust actor system similar to erlang, but we'll probably never get that.

[1] https://github.com/abhinavsingh/proxy.py

• jobserver

  1 8 0.0 Python

  A nestable, single-file Jobserver for CPython 3.5+ which can be copied into any project.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• gevent

  5 6,161 8.7 Python

  Coroutine-based concurrency library for Python

  

I'm not sure how much it replicates the CSP model, but the closest thing I've found to Go-style concurrency in Python is gevent: https://github.com/gevent/gevent

I personally still prefer to use it in all my projects.

• one-ring

  1 1 3.2 Python

  CSP on top of AsyncIO

• stackless

  2 995 0.0 Python

  The Stackless Python programming language

  

Is stackless still an alternative? (It used to be quite hot 1.5 decade ago)

https://github.com/stackless-dev/stackless/wiki/

• unsync

  4 863 0.0 Python

  Unsynchronize asyncio

I will briefly plug my library `unsync` (https://github.com/alex-sherman/unsync#quick-overview) which wraps all these methods (multiprocessing/threading/asyncio) into singular/simple-ish API.

It's a bit overly simple, but it's helped a few times writing code the makes use of several concurrency methods and combining them together etc.

Suggest a related project