leakdice-rust VS librdkafka

Maybe try Leakdice: https://github.com/tialaramex/leakdice in C or rewritten in Rust: https://github.com/tialaramex/leakdice-rust/

In my obviously biased opinion, very specialised, but sometimes exactly what you needed (I have used this in anger maybe 2-3 times in my career since then, which is why I wrote the C version):

https://github.com/tialaramex/leakdice (or https://github.com/tialaramex/leakdice-rust)

Leakdice implements some of Raymond Chen's "The poor man’s way of identifying memory leaks" for you. On Linux at least.

https://bytepointer.com/resources/old_new_thing/20050815_224...

All leakdice does is: You pick a running process which you own, leakdice picks a random heap page belonging to that process and shows you that page as hex + ASCII.

The Raymond Chen article explains why you might ever want to do this.

(or there's a Rust rewrite https://github.com/tialaramex/leakdice-rust because I was learning Rust)

leakdice is not a clever, sophisticated tool like valgrind, or eBPF programming, but that's fine because this isn't a subtle problem - it's very blatant - and running leakdice takes seconds so if it wasn't helpful you've lost very little time.

Here's what leakdice does: It picks a random heap page of a running process, which you suspect is leaking, and it displays that page as ASCII + hex.

That's all, and that might seem completely useless, unless you either read Raymond Chen's "The Old New Thing" or you paid attention in statistics class.

Because your program is leaking so badly the vast majority of heap pages (leakdice counts any pages which are writable and anonymous) are leaked. Any random heap page, therefore, is probably leaked. Now, if that page is full of zero bytes you don't learn very much, it's just leaking blank pages, hard to diagnose. But most often you're leaking (as was happening here) something with structure, and very often sort of engineer assigned investigating a leak can look at a 4kbyte page of structure and go "Oh, I know what that is" from staring at the output in hex + ASCII.

This isn't a silver bullet, but it's very easy and you can try it in like an hour (not days, or a week) including writing up something like "Alas the leaked pages are empty" which isn't a solution but certainly clarifies future results.

recent PR: https://github.com/confluentinc/librdkafka/pull/4275

# Kafka configuration # https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md bootstrap.servers= security.protocol=SASL_SSL sasl.mechanisms=PLAIN sasl.username= sasl.password= compression.type=gzip compression.level=9 statistics.interval.ms=1000

But what about Kafka producer clients in other languages? The excellent librdkafka project is a C/C++ implementation of Kafka clients and is widely used for non-JVM Kafka applications. Additionally, Kafka clients in other languages (Python, C#) build on top of it. The default partitioner for librdkafka uses the CRC32 hash function to get the correct partition for a key.

We could have made some changes at the librdkafka level (see this), but we didn’t really want to pursue this (at least not yet).

You want records with the same key to always land on the same partition, so you need all the clients to use the same hashing algorithm. The easiest way to do that is to make sure the librdkafka client uses the java compatible murmur2_random hash algorithm. See “Partitioner” section here: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md

As my first "real world" (ish) project in Vlang, I'm trying to copy https://github.com/confluentinc/confluent-kafka-go, which is a Go wrapper for Kafka C client library, https://github.com/edenhill/librdkafka

If you're using Kafka in a Node.js app, it's likely that you'll need node-rdkafka. This is a library that wraps the librdkafka library and makes it available in Node.js. According to the project's README, "All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library."

# Parse the configuration. # See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md config_parser = ConfigParser() config_parser.read_file(args.config_file) config = dict(config_parser['default']) # Create Producer instance producer = Producer(config)

• git clone https://github.com/edenhill/librdkafka.git