kafka-connect-jdbc VS kafdrop

Event sourcing is an effective architectural pattern to record changes to the application state. Event sequence is important - we need changes as they were originally applied. incoming events are first persisted into Kafka and then processed by services independently. Kafka, hence, becomes our source of truth (SOT), a data source that gives a complete picture of the data object as a whole. This, however, meant dramatic changes in our core application. Our source of truth is still the core database, but we generate events in Kafka when data gets persisted. To ensure transactional behavior, we employed Transactional Outbox pattern. Essentially, we a) created a new events table in the database, b) wrote event data in the same transaction when we update our SOT table. Kafka Connect is subsequently used to read this table and insert records in relevant Kafka topics. This ensures that we never have an inconsistent situation where data was inserted in the database but the event is not added to Kafka topic or vice versa. We evaluated a few connectors for sourcing data from Mysql (JdbcSourceConnector, and Debezium). Our scenario was supported out of the box in JdbcSourceConnector, making it possible to have one event table in Mysql where different rows could be routed to a relevant topic based on the topic field.

This is different compared to the "polling" technique adopted by the Kafka Connect JDBC connector

git clone https://github.com/obsidiandynamics/kafdrop.git

References [1] Redpanda Console: https://redpanda.com/redpanda-console-kafka-ui [2] Redpanda: https://redpanda.com/ [3] Cluster Deployment of AutoMQ: https://docs.automq.com/zh/docs/automq-opensource/IyXrw3lHriVPdQkQLDvcPGQdnNh [4] Quick Start: https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start [5] Release Redpanda Console: https://github.com/redpanda-data/console/releases/tag/v2.6.0 [6] Redpanda Console Configuration: https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file [7] Kafdrop Github: https://github.com/obsidiandynamics/kafdrop

Are there any comparison to CMAK or Kafdrop ?

For Kafka I really liked using tools like conduktor or Kafdrop where you can basically create a subscription ad look at topics, events and payloads.

Kafdrop - https://github.com/obsidiandynamics/kafdrop

No mentions, and 1 open issue with a question 2) Kafka Drop https://github.com/obsidiandynamics/kafdrop

Kafdrop (https://github.com/obsidiandynamics/kafdrop) is very popular and pretty solid. I'd start there. Kafka itself also comes with some cli tools and you can also extract metrics out of it via JMX/jolokia.