debezium VS RabbitMQ

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.

Although they did not make it into production, I experimented with the RabbitMQ message broker, Python (Django, Flask), Kubernetes + minikube, JWT, and NGINX. This was a hobby project, but I intended to learn about microservices along the way.

Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)

Later, we discovered Propan, a library created by Nikita Pastukhov, which solved similar problems but for RabbitMQ. Recognizing the potential for collaboration, we joined forces with Nikita to build a unified library that could work seamlessly with both Kafka and RabbitMQ. And that's how FastStream came to be—a solution born out of the need for simplicity and efficiency in microservices development.

Inter-Service Communication: Middleware provides communication channels and protocols that enable microservices to communicate with each other. This can include message brokers like RabbitMQ, Apache Kafka, RPC frameworks like gRPC, or RESTful APIs.

This is an implementation of pub/sub. The publisher is on a separate project. The message broker is Azure Service Bus. We use NServiceBus for code implementation. I use rabbitMQ broker for local tests. Nothing I can do about the tech stack. This is more of a high level single project structure review 😅

RabbitMQ is a robust and highly configurable open-source message broker that implements the Advanced Message Queuing Protocol (AMQP).

When using third-party services, especially within a "transaction", it's often a good idea to use a persistent Message Queue (MQ) system like RabbitMQ. Go through all their tutorials to get a really good understanding of how message queues work and how they can be used to solve your problem.

The best technical solution to provide the event queues is to use a message-broker technology like RabbitMQ.