What If We Could Rebuild Kafka from Scratch?

1. fluvio

   1 35 4,957 9.4 Rust

   🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.

   

   https://github.com/infinyon/fluvio

   For the past 2 years we have also been building Flink using Rust and WASM.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. NATS

   2 125 17,482 9.9 Go

   High-Performance server for NATS.io, the cloud and edge native messaging system.

   

   https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

4. debezium

   3 90 11,602 9.9 Java

   Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

   

   > why is there no in-memory kafka server that I can use for simple testing purposes.

   Take a look at Debezium's KafkaCluster, which is exactly that: https://github.com/debezium/debezium/blob/main/debezium-core....

   It's used within Debezium's test suite. Check out the test for this class itself to see how it's being used: https://github.com/debezium/debezium/blob/main/debezium-core...

5. parallel-consumer

   4 9 194 7.0 Java

   Parallel Apache Kafka client wrapper with per message ACK, client side queueing, a simpler consumer/producer API with key concurrency and extendable non-blocking IO processing.

   

   Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer

   It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed.

   Most of the time, it was only processing a small number of records out of order so this bookkeeping was insignificant, but if one key gets stuck, it would scale to at least 100,000 offsets ahead, at which point enough alarms would go off that we would do something. That's definitely a huge improvement to head of line blocking.

6. stateful-dataflow-examples

   5 2 37 8.4 HTML

   Stateful Dataflows tutorials and examples.

   

7. decaton

   6 2 348 5.4 Java

   High throughput asynchronous task processing on Apache Kafka

   

   There's also a similar project from Line https://github.com/line/decaton.

8. embedded-kafka

   7 2 404 8.3 Scala

   A library that provides an in-memory Kafka instance to run your tests against.

   

9. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

Suggest a related project