Dwarf-Based Stack Walking Using eBPF

1. parca-agent

   1 10 623 9.1 Go

   eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

   

   Author here!

   This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis.

   Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0].

   Even if that's not the case, we want to produce profiles that are as accurate as possible :)

   You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1].

   That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch.

   Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work!

   [0]: https://github.com/parca-dev/parca-agent/pull/1058/commits

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. bcc

   2 79 21,361 9.2 C

   BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

   

   That's right! There's no "traditional" loops as programs have to be proved to terminate at some point.

   That being said, very recently support for bounded loops landed [0]. It's very exciting and useful, and I've seen it reduce verification times significantly, but we can't use this yet as it requires kernel 5.3 or greater, and we would like to support as many users as possible!

   [0]: https://lwn.net/Articles/877062/

   [1]: https://github.com/iovisor/bcc/commit/38304256c49a02aecbf78f...

4. scalene

   3 36 12,653 8.7 Python

   Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

   

   This is super awesome work and a great technical explanation of a very deep topic.

   What happens in the case of JIT or FFI? I think I've only ever seen the Python profiler, scalene[0], handle these cases.

   [0]: https://github.com/plasma-umass/scalene

5. cargo-trace

   4 1 38 10.0 Rust

   Flamegraphing tool for perf events

   

   Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools.

   I'm curious what they have to say about complexity/necessity of interpreting all of DWARF. cargo-trace (an neat and conceptually similar but abandoned project, I think) [1] says:

   > It can be empirically determined that almost all dwarf programs consist of a single instruction and use only three different instructions. rip+offset, rsp+offset or *cfa+offset, where cfa is the rsp value of the previous frame. The result of the unwinding is an array of instruction pointers.

   Do you find this to be true? Is more complex interpreting of DWARF necessary?

   And in the lkml thread linked from the article, Linus is extremely pessimistic about DWARF unwinding, [2] I'm sure not without justification. He's talking about kernel stacks, and I think the trade-off is different when you're trying to profile existing userspace applications and libraries compiled and implemented however, but nonetheless I'm curious to hear the authors say how applicable they think his points are.

   [1] https://github.com/dvc94ch/cargo-trace

   [2] https://lkml.org/lkml/2012/2/10/356

6. framehop

   5 1 91 7.5 Rust

   Stack unwinding library in Rust

   

7. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

Suggest a related project