kafka-gitops VS debezium

After I have seen kafka-gitops and JulieOps going "out of business" I am a bit wary of projects with a single maintainer.

cool wasn’t aware of the terraform topic provider, had been looking into using https://github.com/devshawn/kafka-gitops but think will go with terraform

I have never used those scripts. I don't know if it's for experimenting, or for production or something, but for experimenting I would allow Kafka to create the topics for you, and in production, I would use automation like: https://github.com/devshawn/kafka-gitops

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.