Module Linker VS rr

Process recording by time travel debug seems like a good fit for this problem - then you can capture 100% of process execution and then go back and investigate further.

We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website.

For open source, with rr (https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in.

As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.

Have you ever used rr though?

https://github.com/rr-debugger/rr

https://rr-project.org/ for languages that can be debugged by gdb.

I've never used it but sounds like https://rr-project.org/

https://github.com/rr-debugger/rr/issues/2506#issuecomment-2...

I frequently try to mention how Java with JRebel is the closest to the Lisp experience I've found with non-Lisp, it's more dynamic feeling than so-called dynamic languages. Having something like the condition system being ubiquitous would be golden. (I'm aware there is a Java port though I never got around to playing with it and it doesn't solve the problem of other people's code not using it..) My last big job involved a giant app server that would take minutes to restart if you had to do it, JRebel saved so much time by making things much more reloadable including support for a lot of other libraries' quirks and in general a lot of Java-isms like things configured with XML. Looking under the hood at the JVM you can see traces of Lisp everywhere, like class loaders are just (load)s, it's easy to believe the quote about dragging C++ programmers halfway to Lisp.

Then there's things like rr (https://rr-project.org/) that also seem largely ignored by old unix systems people, despite being exactly appropriate for that environment.

Still, having the whole language available via REPL as Lisp does when you hit a break or error makes up for a lot of weaknesses in the rest of the debugging experience.

I haven't met the individuals like taeric but I do find it plausible that something has been lost for developers whose main experience is in highly separated cloud-oriented systems, whether they go as far as micro-services or not. When you don't have full end-to-end debugging and have to correlate everything with trace ids in logs, and also if policies prevent even getting a debugger hook in production, I can see how one would be less motivated to learn about debugging tools to begin with. (On the other hand you're encouraged to have better logging, and often that's enough to figure out a problem, no need to have a running application.)

Yes, io_uring is still not supported due to fundamental issues in the overall rr architecture which my modification does not resolve. My modification only addresses the HW counter requirement of upstream rr and the other core aspects of rr remain the same.

Normal system calls transition to kernel space and return back from kernel space. They will change your program's memory/process state as soon as they complete. This gives rr an easy boundary when it "can do its thing" to record memory/process state changes or insert results (during replay).

When does an io_uring request/response complete ? That's difficult to say. The kernel/userspace when using io_uring communicate with each other by checking a queue head or tail with memory accesses to see if something got added/removed from request/response ring buffer.

Think of io_uring and userspace cooperating via memory. (Yes, sometimes "proper" traditional ring crossing system calls are made but what makes io_uring so fast is communicating via memory and not via system calls most of the time). Anyways all this makes it difficult for rr to intervene on the boundary between kernel and userspace because this boundary is elusive when it comes to io_uring. It cannot be caught by ptrace ! This explanation is simplified of course.

There are some plans to deal with io_uring by rr maintainers https://github.com/rr-debugger/rr/issues/2613

We are also planning to develop a distributed tracing platform, similar to Jaeger and OpenTelemetry, that continuously records the execution of many distributed processes (e.g. micro-services).

Unlike the existing platforms, which capture only message flows and require you to make educated guesses when some anomaly is observed, our system will let you accurately replay the processing code for each message to quickly identify the root cause for the anomaly.

This would rely on our ability to jump to the specific moment in time when a certain incoming message starts being processed. This moment can be identified either by a log line with a specific format or by a call to some special tracking function (e.g. track_incoming_message(request_id)).

For the system languages, the RR[1] recordings try to be practical by capturing only the non-deterministic events in the program execution. You can pair this with a ring buffer that discards the data after a certain retention period.

For the DB backend, we might add some advanced record filtering options.

(But maybe we are misunderstanding the question?)

1: https://rr-project.org/

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

Or on Linux use rr (https://rr-project.org/) or Undo (https://undo.io - disclaimer: I work on this).

These have the advantage that you only need to repro the bug once (just record it in a loop until the bug happens) then debug at your leisure. So even rare bugs are susceptible.

rr and Undo also both have modes for provoking concurrency bugs (Chaos Mode from rr - https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo..., Thread Fuzzing from Undo - https://undo.io/resources/thread-fuzzing-wild/)