kafka-connect-transform-common VS debezium

But I get the following error: "MAP is not supported type", even though I'm sending as a string. Looking at the source code (https://github.com/jcustenborder/kafka-connect-transform-common/blob/master/src/main/java/com/github/jcustenborder/kafka/connect/transform/common/BaseTransformation.java) it blows at line 120, because it thinks it's Map type. I think the issue with this is because the header it not being correctly sent, because Azure EventHubs sends with Amqp protocol that adds a few bytes in the string...

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.