RabbitMQ VS BeanstalkD

Although they did not make it into production, I experimented with the RabbitMQ message broker, Python (Django, Flask), Kubernetes + minikube, JWT, and NGINX. This was a hobby project, but I intended to learn about microservices along the way.

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Later, we discovered Propan, a library created by Nikita Pastukhov, which solved similar problems but for RabbitMQ. Recognizing the potential for collaboration, we joined forces with Nikita to build a unified library that could work seamlessly with both Kafka and RabbitMQ. And that's how FastStream came to be—a solution born out of the need for simplicity and efficiency in microservices development.

Inter-Service Communication: Middleware provides communication channels and protocols that enable microservices to communicate with each other. This can include message brokers like RabbitMQ, Apache Kafka, RPC frameworks like gRPC, or RESTful APIs.

This is an implementation of pub/sub. The publisher is on a separate project. The message broker is Azure Service Bus. We use NServiceBus for code implementation. I use rabbitMQ broker for local tests. Nothing I can do about the tech stack. This is more of a high level single project structure review 😅

RabbitMQ is a robust and highly configurable open-source message broker that implements the Advanced Message Queuing Protocol (AMQP).

When using third-party services, especially within a "transaction", it's often a good idea to use a persistent Message Queue (MQ) system like RabbitMQ. Go through all their tutorials to get a really good understanding of how message queues work and how they can be used to solve your problem.

The best technical solution to provide the event queues is to use a message-broker technology like RabbitMQ.

There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.

https://beanstalkd.github.io/

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.

If you're measuring throughput in jobs/s, use a real work queue.

Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.

Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.

The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:

https://beanstalkd.github.io/

It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.