BeanstalkD VS Gearman

There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.

https://beanstalkd.github.io/

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.

If you're measuring throughput in jobs/s, use a real work queue.

Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.

Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.

The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:

https://beanstalkd.github.io/

It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.

I am a fan of Gearman to schedule and dispatch distributed jobs, Redis as a collaborative blackboard, and GlusterFS to share models across multiple systems and make bulk data available across the entire system (usually referenced in the blackboard as a pathname).

At work we typically use Gearman (http://gearman.org/) or Symfony messenger (https://symfony.com/doc/current/messenger.html) to queue up a batch of jobs. And then we use supervisord (http://supervisord.org/) to keep a pool of PHP processes running to process the jobs.