rr VS rustfix

We are also planning to develop a distributed tracing platform, similar to Jaeger and OpenTelemetry, that continuously records the execution of many distributed processes (e.g. micro-services).

Unlike the existing platforms, which capture only message flows and require you to make educated guesses when some anomaly is observed, our system will let you accurately replay the processing code for each message to quickly identify the root cause for the anomaly.

This would rely on our ability to jump to the specific moment in time when a certain incoming message starts being processed. This moment can be identified either by a log line with a specific format or by a call to some special tracking function (e.g. track_incoming_message(request_id)).

For the system languages, the RR[1] recordings try to be practical by capturing only the non-deterministic events in the program execution. You can pair this with a ring buffer that discards the data after a certain retention period.

For the DB backend, we might add some advanced record filtering options.

(But maybe we are misunderstanding the question?)

1: https://rr-project.org/

https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

Or on Linux use rr (https://rr-project.org/) or Undo (https://undo.io - disclaimer: I work on this).

These have the advantage that you only need to repro the bug once (just record it in a loop until the bug happens) then debug at your leisure. So even rare bugs are susceptible.

rr and Undo also both have modes for provoking concurrency bugs (Chaos Mode from rr - https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mo..., Thread Fuzzing from Undo - https://undo.io/resources/thread-fuzzing-wild/)

> IntelliTrace is one that comes to mind - there’s nothing remotely close to it’s snapshot debugging that I’ve seen anywhere else, and I’ve really looked.

https://rr-project.org/

Hi, in case you're not already aware of the name clash, there's already a `rr` in the programming world. It's "record and replay": https://rr-project.org/.

Very different, but a very fine tool tool.

I don't know, however a key element is:

> Record/replay system is based on saving and replaying non-deterministic events

> The following non-deterministic data from peripheral devices is saved into the log: mouse and keyboard input, network packets, audio controller input, serial port input, and hardware clocks (they are non-deterministic too, because their values are taken from the host machine). Inputs from simulated hardware, memory of VM, software interrupts, and execution of instructions are not saved into the log, because they are deterministic and can be replayed by simulating the behavior of virtual machine starting from initial state.

So, it's probably not much, you can probably comfortably save minutes of qemu sessions.

Also note the existence of the rr debugger [1], which allows you to reverse debug applications with a ~10% performance hit while recording. To achieve this, it records results of syscalls (only). It will serialize thread events, so have the effect of running applications like on a single core CPU.

[1] https://rr-project.org/

rr.

rr rr rr rr.

rr!

When testing "algorithmic" modules like compilers, it's basically a productivity cheat code to run the tests under https://rr-project.org/. Doing so allows you to deterministically replay execution, seeking forwards and backwards in the timeline of your program's execution, and quickly locate what went wrong in any computation.

For example, if we have

    struct CircleDescription {

It says on https://rr-project.org/ that it supports Go programm, what's the status?

This was because no matter how hard I tried, I kept running into variables that I was watching that appeared to be the same reference, have different values (so I know I was referring to something wrong, but could not for the life of me figure it out, and though if I only had a way to step/trace and do this visually like an AST that highlights changed values and you could see back to its root where it's actually being modified visually)

I found out there's a thing called "deterministic debugging" (and the biggest known example afaict is rr: https://rr-project.org/

Apparently MS has a time-travel debugger... I pictured the AST being populated and repopulated via the steps, and the ability to diff changes over time.

Here's a wiki on various systems, though most of these seem to be typical text based "trace" options.

https://en.wikipedia.org/wiki/Time_travel_debugging