kop VS debezium

I process hospitality data somewhat similarly, but use Pulsar and can individually acknowledge messages, have DLQs built in, and if needed can stream just like Kafka. And if I need Kafka compatibility, I can use something like StreamNative's KOP and get Kafka compatibility over my existing Pulsar queues.

While Pulsar has its own protocol that handles streaming, queues, and a lot of the other features (distributed transactions, functions, etc.), it can also speak other messaging protocols via plug-ins. Looking around, the ones that appear to be actively developed are MQTT, Kafka (so your existing applications that use Kafka can also use Pulsar), AMQP, and JMS.

git clone https://github.com/streamnative/kop.git cd kop

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.