Apache Ignite VS franz-go

Ignite works as you describe:

https://ignite.apache.org/

I wouldn't really recommend this approach, I would think more in terms of subscriptions and topics and less of a 'database'.

Apache Ignite https://ignite.apache.org/

Last days, I started using Apache Ignite as a cache strategy for some applications. Apache Ignite is an open-source In-Memory Data Grid, distributed database, caching, and high-performance computing platform.

Thankfully, Kafka and OpenSearch provide client libraries for a variety of programming languages which make it possible to write your own integration layer. This is exactly what's covered in this blog! We will make use of a custom Go application to ingest data using Go clients for Kafka and OpenSearch.

You might want to gradually replace that one with https://github.com/twmb/franz-go because Shopify is looking to find a new owner for Sarama and, until or if they do, it seems to be falling behind with maintenance: https://github.com/Shopify/sarama/issues/2461 For example, they still haven’t addressed this breaking change https://github.com/Shopify/sarama/issues/2358. franz-go has worked well so far in Benthos https://github.com/benthosdev/benthos/tree/main/internal/impl/kafka and it will likely end up as the only implementation once the Sarama-based one will be deprecated

Can you post the code for that? Franz-go itself publishes benchmark samples for all clients here, so easy to run on your own machine and also review/adapt settings as needed: https://github.com/twmb/franz-go/tree/master/examples/bench

The MSK producer and consumer app use the franz-go library (it also supports MSK IAM authentication).

How to use the franz-go Go Kafka client to connect to MSK Serverless using IAM authentication.

Franz is a fairly obvious name for a Kafka related tool/library so hopefully it won't get confused with https://github.com/twmb/franz-go

github.com/twmb/franz-go -> also early days but this is looking like a fantastic option for a kafka client library if you fancy being an early adopter. I've done the rounds on many kafka client libraries and they always seem to be a harsh compromise in some form or another, but I feel good about this one

If you find the kafka input slow, try kafka_franz. It might be a bit faster, since it’s based on https://github.com/twmb/franz-go. The kafka one is based on https://github.com/Shopify/sarama. You can also write a custom input based on https://github.com/confluentinc/confluent-kafka-go, but this library relies on CGo, which can be annoying.

Redpanda isn’t implemented in Go, but I second its use over the Apache server. Plenty of clients out there in pure go. https://github.com/twmb/franz-go is a personal favorite