Bringing Record and Replay debugging everywhere on Linux

1. rr.soft

   1 4 82 9.6 C++

   Run the rr debugger in a Linux VM on apple silicon macOS or in the cloud. Use the rr debugger without HW performance counters on x86-64 and aarch64.

   

   Author here. Yes there are some tradeoffs:

   - Performance: slower than using rr with HW counters. Both dynamic and static instrumentation employed by _Software Counter mode_ rr slow things down

   - Potential fragility: Dynamic and static instrumentation can make often make record/replay a bit more fragile

   - Currently only x86-64 support has been publicly released. I have aarch64 support working reasonably well internally and it allows me for instance run to rr in a Linux VM running on macOS ! I have yet to figure out my strategy for the aarch64 release so watch https://github.com/sidkshatriya/rr.soft for any updates.

   - Currently can run only on a few recent Linux distributions (e.g. Fedora 40/41, Debian Unstable, Ubuntu 24.10) because it relies on robust debuginfod support that is not widespread yet. See https://github.com/sidkshatriya/rr.soft/wiki#how-does-softwa... for why debuginfod is required. The debuginfod requirement may relaxed in the future with more work

   I would love it if more people tried this out and let me know if things worked out well for them (or not) with their programs.

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. me

   2 18 165 7.9

   This is a space where I express my thoughts and opinions on mostly technical things (by sidkshatriya)

   

4. Dontbug Debugger

   3 1 713 0.0 Go

   rr has always worked with Python in the sense that it can record and replay Python programs.

   However, when you try to debug the program you can only debug the C code the Python interpreter is written in.

   I suppose you want to be able to debug the Python code itself. Here is something that could do this https://pypy.org/posts/2016/07/reverse-debugging-for-python-... . I don't think the project is active nowadays though. Also I haven't used it so can't say whether it is good or not.

   It should be possible to built a Python reverse debugger on top of rr. I know this should be possible because I built something for PHP https://github.com/sidkshatriya/dontbug .

   There are other fancy (and possibly better) things that are possible -- instead of building a Python debugger atop rr you can record the full trace of the Python program in one go e.g. record the values of important variables at each executed line of the Python program. This is an area which I've done some work internally but nothing public released yet :-) !

5. rr

   4 122 9,939 9.4 C++

   Record and Replay Framework

   

   Yes, io_uring is still not supported due to fundamental issues in the overall rr architecture which my modification does not resolve. My modification only addresses the HW counter requirement of upstream rr and the other core aspects of rr remain the same.

   Normal system calls transition to kernel space and return back from kernel space. They will change your program's memory/process state as soon as they complete. This gives rr an easy boundary when it "can do its thing" to record memory/process state changes or insert results (during replay).

   When does an io_uring request/response complete ? That's difficult to say. The kernel/userspace when using io_uring communicate with each other by checking a queue head or tail with memory accesses to see if something got added/removed from request/response ring buffer.

   Think of io_uring and userspace cooperating via memory. (Yes, sometimes "proper" traditional ring crossing system calls are made but what makes io_uring so fast is communicating via memory and not via system calls most of the time). Anyways all this makes it difficult for rr to intervene on the boundary between kernel and userspace because this boundary is elusive when it comes to io_uring. It cannot be caught by ptrace ! This explanation is simplified of course.

   There are some plans to deal with io_uring by rr maintainers https://github.com/rr-debugger/rr/issues/2613

6. nixseparatedebuginfod

   5 2 96 7.6 Rust

   Downloads and provides debug symbols and source code for nix derivations to gdb and other debuginfod-capable debuggers as needed.

   

   Probably not at the moment.

   _Software Counter mode_ rr requires robust support for debuginfod in your Linux distribution.

   NixOS to my knowledge does not provide debuginfo via a dedicated debuginfod server for all packages so this will make things unreliable for debugging via _Software Counters mode_ rr for packages for which debuginfo is not available via debuginfod.

   I recently learned about https://github.com/symphorien/nixseparatedebuginfod for nix (note: I haven't used this so I don't know how reliable or good this is). Anyways, this project requires setting `separateDebuginfo = true;` for the derivation for the debuginfo to be available via debuginfod. This is an opt in approach but we need pervasive support like it exists in Fedora (and some others).

7. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

Suggest a related project