cainophile VS debezium

If you want to listen to database changes in Elixir you can also get really good stuff done by using Cainophile (https://github.com/cainophile/cainophile). Same mechanism. I don't know the details of Debezium so I can't say if you are leaving fantastic things on the table. But I've had good fun with Cainophile. For example I've used it in my videos on Electric SQL to react to changes in a Postgres database. It matches nicely with realtime-ish UI via LiveView. So meshes really good with the current Elixir stack.

https://github.com/cainophile/cainophile

I happened to do a similar thing but I adapted cainophile into an Elixir “OffBroadway” producer:

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.