confluent-kafka-go
leakdice
confluent-kafka-go | leakdice | |
---|---|---|
1 | 4 | |
1 | 18 | |
- | - | |
0.0 | 0.0 | |
about 1 year ago | about 5 years ago | |
Go | C | |
Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
confluent-kafka-go
-
Hunting down a C memory leak in a Go program
So, in the interests of full transparency - we at Zendesk are actually running a fork of confluent-kafka-go, which I forked to add, amongst other things, context support: https://github.com/confluentinc/confluent-kafka-go/pull/626
This bug actually happened because I mis-merged upstream into our fork and missed an important call to rd_kafka_poll_set_consumer: https://github.com/zendesk/confluent-kafka-go/commit/6e2d889...
leakdice
-
My Rust program (Well, game) is leaking memory, 4MB/s.
Maybe try Leakdice: https://github.com/tialaramex/leakdice in C or rewritten in Rust: https://github.com/tialaramex/leakdice-rust/
-
Twenty Years of Valgrind
In my obviously biased opinion, very specialised, but sometimes exactly what you needed (I have used this in anger maybe 2-3 times in my career since then, which is why I wrote the C version):
https://github.com/tialaramex/leakdice (or https://github.com/tialaramex/leakdice-rust)
Leakdice implements some of Raymond Chen's "The poor man’s way of identifying memory leaks" for you. On Linux at least.
https://bytepointer.com/resources/old_new_thing/20050815_224...
All leakdice does is: You pick a running process which you own, leakdice picks a random heap page belonging to that process and shows you that page as hex + ASCII.
The Raymond Chen article explains why you might ever want to do this.
-
Hunting down a C memory leak in a Go program
(or there's a Rust rewrite https://github.com/tialaramex/leakdice-rust because I was learning Rust)
leakdice is not a clever, sophisticated tool like valgrind, or eBPF programming, but that's fine because this isn't a subtle problem - it's very blatant - and running leakdice takes seconds so if it wasn't helpful you've lost very little time.
Here's what leakdice does: It picks a random heap page of a running process, which you suspect is leaking, and it displays that page as ASCII + hex.
That's all, and that might seem completely useless, unless you either read Raymond Chen's "The Old New Thing" or you paid attention in statistics class.
Because your program is leaking so badly the vast majority of heap pages (leakdice counts any pages which are writable and anonymous) are leaked. Any random heap page, therefore, is probably leaked. Now, if that page is full of zero bytes you don't learn very much, it's just leaking blank pages, hard to diagnose. But most often you're leaking (as was happening here) something with structure, and very often sort of engineer assigned investigating a leak can look at a 4kbyte page of structure and go "Oh, I know what that is" from staring at the output in hex + ASCII.
This isn't a silver bullet, but it's very easy and you can try it in like an hour (not days, or a week) including writing up something like "Alas the leaked pages are empty" which isn't a solution but certainly clarifies future results.
-
`Zig Cc`: A Powerful Drop-In Replacement for GCC/Clang
Right. Even in an entirely safe language you can have leaks, and valgrind is an effective way to find those leaks if you can afford the virtualisation overhead.
If you can't afford the virtualisation overhead, and you need to find leaks you should try what Raymond Chen suggests in "The poor man's way of identifying memory leaks" (not bothering to link since Microsoft will only move it anyway, they have several times since I read it). If you are too lazy to do it by hand, or find the technique works but wish it less manual, this is what Leakdice does:
https://github.com/tialaramex/leakdice
What are some alternatives?
librdkafka - The Apache Kafka C/C++ library
libclang_rt.builtins-wasm32.a - The missing libclang_rt.builtins-wasm32.a file to compile to WebAssembly.
kafka-go - Kafka library in Go
mevi - A memory visualizer in Rust (ptrace + userfaultfd)
bytehound - A memory profiler for Linux.
hotspot - The Linux perf GUI for performance analysis.
heaptrack - A heap memory profiler for Linux
Confluent Kafka Golang Client - Confluent's Apache Kafka Golang client
leakdice-rust - Rust re-implementation of leakdice