BeanstalkD VS ZeroMQ

There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.

https://beanstalkd.github.io/

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.

If you're measuring throughput in jobs/s, use a real work queue.

Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.

Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.

The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:

https://beanstalkd.github.io/

It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.

Slightly OT:

Are ZeroMQ and NanoMQ still widely used (and recommended)?

https://github.com/zeromq/libzmq

https://github.com/nanomq/nanomq

Remarkable, up until recently, requests for a new release were sumewhat brusquely rejected and marked as spam.

https://github.com/zeromq/libzmq/issues/4455

I wonder what made the maintainer change his mind.

FP Libraries: gRPC, ZeroMQ, and AREG are examples of libraries with a special focus on providing possibilities for Interprocess Communication. Developed using C++, they facilitate communication through predefined APIs, emphasizing functional programming concepts.

Unlikely, but they seem to be different things altogether. BlazingMQ appears to be a traditional message queue (think ActiveMQ), with message peristence. ZeroMQ is more of a network middleware (think Tibco Rendezvous), and does not include persistence.

BlazingMQ also appears to be more of a "platform" or "service" that an app can use (sort of like Oracle, say) -- ZeroMQ includes libraries that one can use to build an app, service or platform, but none is provided "out of the box".

Which makes it harder to get started with ZeroMQ, since by definition every ZeroMQ app is essentially built "from scratch".

If you're interested in ZeroMQ, you may want to check out OZ (https://github.com/nyfix/OZ), which is a Rendezvous-like platform that uses the OpenMAMA API (https://github.com/finos/OpenMAMA) and ZeroMQ (https://github.com/zeromq/libzmq) transport to provide a full-featured network middleware implementation. OZ has been used in our shop since 2020 handling approx 50MM high-value messages per day on our global FIX network.

-- Using src='https://github.com/zeromq/libzmq/releases/download/v4.3.4/zeromq-4.3.4.tar.gz'

Just check copying file in source repo https://github.com/zeromq/libzmq

I think the situation is more subtle than the poster admits.

No, ZeroMQ and successors do not tell you about socket state. You can't detect disconnection or reconnection. But then if a TCP connection fails in some way that does not lead to disconnection (packets getting dropped, remote machine powers down), it can't possibly tell you about that either, but you still need to deal with it. So in any case, you need some sort of application-level error detection and recovery; you need heartbeats, and serial numbers in messages, and a protocol for explicitly restarting a connection and performing the initial handshake. And once you have that, explicit connection events from ZeroMQ are much less important.

Admittedly, given that this is a TCP transport, reporting reconnections would still be useful, because TCP won't ever drop messages from the interior of a sequence itself (if it delivers 15, it has delivered 1 - 14 already), so you shouldn't need the serial numbers.

And if it's really not possible to detect authentication failures, than that seems rubbish. And it seems that is indeed the case: https://github.com/zeromq/libzmq/issues/3505

From https://github.com/zeromq/libzmq