Dwarf-Based Stack Walking Using eBPF

• parca-agent

  10 484 9.9 Go

  eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

  

Author here!

This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis.

Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0].

Even if that's not the case, we want to produce profiles that are as accurate as possible :)

You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1].

That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch.

Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work!

[0]: https://github.com/parca-dev/parca-agent/pull/1058/commits

• bcc

  71 19,499 9.2 C

  BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

  

That's right! There's no "traditional" loops as programs have to be proved to terminate at some point.

That being said, very recently support for bounded loops landed [0]. It's very exciting and useful, and I've seen it reduce verification times significantly, but we can't use this yet as it requires kernel 5.3 or greater, and we would like to support as many users as possible!

[0]: https://lwn.net/Articles/877062/

[1]: https://github.com/iovisor/bcc/commit/38304256c49a02aecbf78f...

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• scalene

  32 11,174 9.2 Python

  Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

  

This is super awesome work and a great technical explanation of a very deep topic.

What happens in the case of JIT or FFI? I think I've only ever seen the Python profiler, scalene[0], handle these cases.

[0]: https://github.com/plasma-umass/scalene

• cargo-trace

  1 35 10.0 Rust

  Flamegraphing tool for perf events

Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools.

I'm curious what they have to say about complexity/necessity of interpreting all of DWARF. cargo-trace (an neat and conceptually similar but abandoned project, I think) [1] says:

> It can be empirically determined that almost all dwarf programs consist of a single instruction and use only three different instructions. rip+offset, rsp+offset or *cfa+offset, where cfa is the rsp value of the previous frame. The result of the unwinding is an array of instruction pointers.

Do you find this to be true? Is more complex interpreting of DWARF necessary?

And in the lkml thread linked from the article, Linus is extremely pessimistic about DWARF unwinding, [2] I'm sure not without justification. He's talking about kernel stacks, and I think the trade-off is different when you're trying to profile existing userspace applications and libraries compiled and implemented however, but nonetheless I'm curious to hear the authors say how applicable they think his points are.

[1] https://github.com/dvc94ch/cargo-trace

[2] https://lkml.org/lkml/2012/2/10/356

• framehop

  1 74 8.1 Rust

  Stack unwinding library in Rust

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project