raft VS Apache Kafka

etcd Raft — a library for maintaining replicated state machines (written in Go);

Have you seen this implementation? https://github.com/etcd-io/raft. I’ve used this as a starting point for our consensus with various services.

This is usual in parallel and distributed systems. For k8s, I recommend you either use Apache Zookeeper to coordinate “logical locks” for operations across pods or use the RAFT consensus protocol for scenarios where you need coordination amongst pods. For context, k8s itself uses the RAFT consensus protocol for etcd; see https://github.com/etcd-io/raft

Apache Kafka — a distributed event streaming platform implementing a variant of the Raft consensus protocol (written in Java, integrated with Scala);

Spring does this during autoconfiguration. For example this and this. When the user adds a configuration then it gets to overwrite the default from the template. I am looking for something similar, perhaps simpler approach.

Source and documentation on GitHub

Code is still in trunk though. https://github.com/apache/kafka/tree/trunk/connect/file/src/main/java/org/apache/kafka/connect/file

I briefly described the algorithm in this article and there is a wonderful article from Kafka that goes into more depth in their general purpose implementation. My implementation is specialized and over optimized in comparison, e.g. by using bit manipulation to avoid more expensive division/modulus instructions. Tokio rewrote their timerwheel after I showed them mine, borrowing some ideas but also staying more general. Hope that helps!

Apache Kafka