Dwarf-Based Stack Walking Using eBPF

• parca-agent

  5 295 7.2 Go

  eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

  

  Author here!

  This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis.

  Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0].

  Even if that's not the case, we want to produce profiles that are as accurate as possible :)

  You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1].

  That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch.

  Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work!

  [0]: https://github.com/parca-dev/parca-agent/pull/1058/commits

• bcc

  25 16,869 8.8 C

  BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

  

  That's right! There's no "traditional" loops as programs have to be proved to terminate at some point.

  That being said, very recently support for bounded loops landed [0]. It's very exciting and useful, and I've seen it reduce verification times significantly, but we can't use this yet as it requires kernel 5.3 or greater, and we would like to support as many users as possible!

  [0]: https://lwn.net/Articles/877062/

  [1]: https://github.com/iovisor/bcc/commit/38304256c49a02aecbf78f...

• InfluxDB

  www.influxdata.com

  sponsored

  Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  

• scalene

  17 7,501 9.0 JavaScript

  Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

  

  This is super awesome work and a great technical explanation of a very deep topic.

  What happens in the case of JIT or FFI? I think I've only ever seen the Python profiler, scalene[0], handle these cases.

  [0]: https://github.com/plasma-umass/scalene

• cargo-trace

  1 31 10.0 Rust

  Flamegraphing tool for perf events

  Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools.

  I'm curious what they have to say about complexity/necessity of interpreting all of DWARF. cargo-trace (an neat and conceptually similar but abandoned project, I think) [1] says:

  > It can be empirically determined that almost all dwarf programs consist of a single instruction and use only three different instructions. rip+offset, rsp+offset or *cfa+offset, where cfa is the rsp value of the previous frame. The result of the unwinding is an array of instruction pointers.

  Do you find this to be true? Is more complex interpreting of DWARF necessary?

  And in the lkml thread linked from the article, Linus is extremely pessimistic about DWARF unwinding, [2] I'm sure not without justification. He's talking about kernel stacks, and I think the trade-off is different when you're trying to profile existing userspace applications and libraries compiled and implemented however, but nonetheless I'm curious to hear the authors say how applicable they think his points are.

  [1] https://github.com/dvc94ch/cargo-trace

  [2] https://lkml.org/lkml/2012/2/10/356

• framehop

  1 48 10.0 Rust

  Stack unwinding library in Rust

• SonarQube

  www.sonarqube.org

  sponsored

  Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  

Suggest a related project