Dwarf-Based Stack Walking Using eBPF

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
  • parca-agent

    eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!

  • Author here!

    This project looks interesting and as you mention it seems conceptually similar! I would agree with that quote, generally speaking, but I think it really depends on a binary to binary basis.

    Some binaries outsource most of the work to dynamic libraries. Unfortunately, DWARF expressions are typically emitted for these program counter ranges, so it's desirable to at least implement a subset of expressions [0].

    Even if that's not the case, we want to produce profiles that are as accurate as possible :)

    You are totally right, they were discussing kernel stacks where stakes are higher as it needs to work perfectly, otherwise kernel live patching would not reliably work, among others. The kernel has now some unwind table format, that can be used in x86_64, called ORC [1].

    That being said the parser for DWARF would still have to live in the kernel, and I am not sure if kernel devs would like to accept such a patch.

    Ideally we would transition to an unwind-specific format for user-space (something like ours, for example) and perhaps have a suitable unwinder in the kernel, rather than having to implement it in a BPF program. This is something we are considering for the future, but it's not free from problems (increased executable size, redundant unwind information, etc). But this is exactly why we wanted to have a conversation with the communities interested in this work!

    [0]: https://github.com/parca-dev/parca-agent/pull/1058/commits

  • bcc

    BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more

  • That's right! There's no "traditional" loops as programs have to be proved to terminate at some point.

    That being said, very recently support for bounded loops landed [0]. It's very exciting and useful, and I've seen it reduce verification times significantly, but we can't use this yet as it requires kernel 5.3 or greater, and we would like to support as many users as possible!

    [0]: https://lwn.net/Articles/877062/

    [1]: https://github.com/iovisor/bcc/commit/38304256c49a02aecbf78f...

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • scalene

    Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

  • This is super awesome work and a great technical explanation of a very deep topic.

    What happens in the case of JIT or FFI? I think I've only ever seen the Python profiler, scalene[0], handle these cases.

    [0]: https://github.com/plasma-umass/scalene

  • cargo-trace

    Flamegraphing tool for perf events

  • Are the authors here? Thanks for this! I'm always thrilled to see advances in profiling tools.

    I'm curious what they have to say about complexity/necessity of interpreting all of DWARF. cargo-trace (an neat and conceptually similar but abandoned project, I think) [1] says:

    > It can be empirically determined that almost all dwarf programs consist of a single instruction and use only three different instructions. rip+offset, rsp+offset or *cfa+offset, where cfa is the rsp value of the previous frame. The result of the unwinding is an array of instruction pointers.

    Do you find this to be true? Is more complex interpreting of DWARF necessary?

    And in the lkml thread linked from the article, Linus is extremely pessimistic about DWARF unwinding, [2] I'm sure not without justification. He's talking about kernel stacks, and I think the trade-off is different when you're trying to profile existing userspace applications and libraries compiled and implemented however, but nonetheless I'm curious to hear the authors say how applicable they think his points are.

    [1] https://github.com/dvc94ch/cargo-trace

    [2] https://lkml.org/lkml/2012/2/10/356

  • framehop

    Stack unwinding library in Rust

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Scalene: A high-performance CPU GPU and memory profiler for Python

    1 project | /r/hypeurls | 18 Jun 2023
  • Scalene: A high-performance, CPU, GPU, and memory profiler for Python

    1 project | news.ycombinator.com | 18 Jun 2023
  • How can I find out why my python is so slow?

    2 projects | /r/Python | 30 May 2023
  • How to troubleshoot memory leaks in Go with Grafana Pyroscope

    1 project | /r/golang | 19 Apr 2023
  • Cum as putea sa imbunatatesc timpul de rulare al pitonului?

    1 project | /r/programare | 14 Mar 2023