Pact JVM VS debezium

API Contract Testing might be worth exploring. Gives you the ability to run it quickly locally or in CI/CD. A centralized way to validate if your changes broke another service might also be interesting. That series of videos from the same team is gold: https://docs.pact.io

We use Pact for contract testing https://docs.pact.io/

The introduction here is quite good: https://docs.pact.io/

Pact.Net documentation: https://docs.pact.io/ Mountebank documentation: http://www.mbtest.org/ The Microsoft.AspNetCore.TestHost package on GitHub: https://github.com/aspnet/AspNetCore/tree/main/src/Testing/Microsoft.AspNetCore.TestHost

There's also https://docs.pact.io/ but I haven't had time yet to really look into it but probably someone else can tell something about it.

This PoC shows a step by step implementation of contract testing using Pact.

Contract tests assert that inter-application messages conform to a shared understanding documented in a contract. Without contract testing, the only way to ensure that applications will work correctly together is by using expensive and brittle integration tests.

If you are updating your API based on user feedback, you can work with your team or stakeholders to build an API contract describing how your API should function. You can use JSON schemas to document your API’s endpoint response codes and set up the contract. The contract must be accepted by all parties. Then, any changes made to the API must pass the contract test before being approved. Contract testing can be automated and incorporated into your CI/CD pipeline. Contract testing tools include Pact and Spring Cloud Contract.

Pact.

I have contract tests (https://docs.pact.io/) to ensure the integration works.

They manage data in the application layer and your original data stays where it is. This way data consistency is no longer an issue as it was with streaming databases. You can use Change Data Capture (CDC) services like Debezium by directly connecting to your primary database, doing computational work, and saving the result back or sending real-time data to output streams.

Both of these articles mention a key player, Debezium. In fact, Debezium has had a place in the modern infrastructure. Let's use a diagram to understand why.

This might be overkill, but you could use an extra tool like https://debezium.io to capture logs about all creates, updates, and deletes in your table

If you're considering a relational database for an event-driven architecture, check out Debezium. It lets you stream changes to relational databases, and subscribe to change events.

Debezium

https://debezium.io/ comes to mind as an open source product, but there are a gazillion of these tools out there.

Contrary to what it sounds, it’s quite straightforward. The database changes are captured via Debezium and published as events on Apache Kafka. ClickHouse consumes those changes in partial order by Kafka Engine. Real-time and eventually consistent.