confluent-kafka-go VS leakdice

So, in the interests of full transparency - we at Zendesk are actually running a fork of confluent-kafka-go, which I forked to add, amongst other things, context support: https://github.com/confluentinc/confluent-kafka-go/pull/626

This bug actually happened because I mis-merged upstream into our fork and missed an important call to rd_kafka_poll_set_consumer: https://github.com/zendesk/confluent-kafka-go/commit/6e2d889...

Maybe try Leakdice: https://github.com/tialaramex/leakdice in C or rewritten in Rust: https://github.com/tialaramex/leakdice-rust/

In my obviously biased opinion, very specialised, but sometimes exactly what you needed (I have used this in anger maybe 2-3 times in my career since then, which is why I wrote the C version):

https://github.com/tialaramex/leakdice (or https://github.com/tialaramex/leakdice-rust)

Leakdice implements some of Raymond Chen's "The poor man’s way of identifying memory leaks" for you. On Linux at least.

https://bytepointer.com/resources/old_new_thing/20050815_224...

All leakdice does is: You pick a running process which you own, leakdice picks a random heap page belonging to that process and shows you that page as hex + ASCII.

The Raymond Chen article explains why you might ever want to do this.

(or there's a Rust rewrite https://github.com/tialaramex/leakdice-rust because I was learning Rust)

leakdice is not a clever, sophisticated tool like valgrind, or eBPF programming, but that's fine because this isn't a subtle problem - it's very blatant - and running leakdice takes seconds so if it wasn't helpful you've lost very little time.

Here's what leakdice does: It picks a random heap page of a running process, which you suspect is leaking, and it displays that page as ASCII + hex.

That's all, and that might seem completely useless, unless you either read Raymond Chen's "The Old New Thing" or you paid attention in statistics class.

Because your program is leaking so badly the vast majority of heap pages (leakdice counts any pages which are writable and anonymous) are leaked. Any random heap page, therefore, is probably leaked. Now, if that page is full of zero bytes you don't learn very much, it's just leaking blank pages, hard to diagnose. But most often you're leaking (as was happening here) something with structure, and very often sort of engineer assigned investigating a leak can look at a 4kbyte page of structure and go "Oh, I know what that is" from staring at the output in hex + ASCII.

This isn't a silver bullet, but it's very easy and you can try it in like an hour (not days, or a week) including writing up something like "Alas the leaked pages are empty" which isn't a solution but certainly clarifies future results.

Right. Even in an entirely safe language you can have leaks, and valgrind is an effective way to find those leaks if you can afford the virtualisation overhead.

If you can't afford the virtualisation overhead, and you need to find leaks you should try what Raymond Chen suggests in "The poor man's way of identifying memory leaks" (not bothering to link since Microsoft will only move it anyway, they have several times since I read it). If you are too lazy to do it by hand, or find the technique works but wish it less manual, this is what Leakdice does:

https://github.com/tialaramex/leakdice