tqs VS BeanstalkD

I wrote https://github.com/TinyWebServices/tqs a couple of years ago. It is modelled after SQS and runs in a single threaded Tornado server.

I don’t know how many messages per second it does but for a podcast crawling side project I have processed billions of messages through this little Python wrapper around SQLite. Zero problems. It just keeps running happily.

There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.

https://beanstalkd.github.io/

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.

If you're measuring throughput in jobs/s, use a real work queue.

Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.

Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.

The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:

https://beanstalkd.github.io/

It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.