arq VS sdk-python

I need to process a lot of long running IO heavy jobs with background workers. I've been using ARQ for a while but decided to take a crack at writing my own distributed queue.

Lately I've been writing asynchronous python code and yes, the resource integration problem has come again. Because now from version 1.4 SQLAlchemy has become asynchronous a new boilerplate was created. Now, as a scheduler, I took a completely asynchronous Arq. Considering the specifics of the service, long I/O operations, it seems that the service turned out to be more optimal in asynchronous execution. I haven't measured the performance yet, but I think I'll write another post about it.

My problem with Temporal is that it doesn't support gevent https://github.com/temporalio/sdk-python/issues/59

> Unless Pydantic is downloading all OS binaries with the package and loading the right one at runtime, this would become a "problem" as well.

Nah, it's not that bad. I built a Rust-backed Python library used by many [0], and with setuptools-rust (maturin wasn't flexible enough at the time) and cibuildwheel and GH actions, the wheels are built/shipped with the shared libraries embedded and the end user never has to worry or even be aware of its presence.

Pydantic has already been shipping a binary mode with an option for pure Python, so maybe they'll keep the pure Python mode around.

0 - https://github.com/temporalio/sdk-python

Either way, let us know how it goes! Building something cool? We’d love to hear about it! Our forum has a new Show & Tell section. If you want to share, we’ll send you some sweet swag. Have feedback on how we can do better? We want to know that too. Raise an issue in the SDK or samples repos, or send us an email ([email protected]).

We built to logic backing the Temporal Python SDK[0] in Rust and leverage PyO3. Unfortunately Maturin didn't let us do some of the advanced things we needed to do for wheel creation (at the time, unsure now), so we use setuptools-rust with Poetry.

0 - https://github.com/temporalio/sdk-python

Going to shamelessly plug Temporal’s Python SDK which was designed for asyncio.

https://github.com/temporalio/sdk-python

Disclaimer: I work for Temporal

hey sorry for taking a while to respond, was busy with personal stuff and hope you see this.

1. yes. its on the order of months. start watching https://github.com/temporalio/sdk-python

2. temporal itself will not produce something like that, because we much rather have a lively community of third party maintainers/startups do that and be their supporters rather than competitors. interested?