maxwell VS debezium

we started using https://github.com/zendesk/maxwell its not perfect, e.g. its difficult to know who changed what, but we solved this by adding the the user performing the change to the mixin

You could consider some CDC software, which logs database queries (inserts, updates and deletes). For example maxwells daemon https://maxwells-daemon.io/

Or you could generate a Kafka queue from MariaDB directly with Maxwell.

Most of the solutions out there for doing CDC on MySQL involve using the transaction log. These include https://github.com/debezium/debezium , https://github.com/airbnb/SpinalTap, and https://github.com/zendesk/maxwell.

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.