dettrace
shadow
dettrace | shadow | |
---|---|---|
2 | 11 | |
29 | 1,352 | |
- | 1.0% | |
2.6 | 9.8 | |
over 3 years ago | 9 days ago | |
C++ | Rust | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dettrace
-
Deterministic Linux for Controlled Testing and Software Bug-Finding
Note that this is a follow-on project from the earlier Dettrace system, which was applied mainly to reproducible builds (as in the academic paper, https://dl.acm.org/doi/10.1145/3373376.3378519, and presented to the Debian Reproducible Builds summit):
- https://github.com/dettrace/dettrace
And one cool part of it is this Rust program instrumentation layer:
- https://github.com/facebookexperimental/reverie
It's good for building OS-emulator style projects or tracing tools.
-
Shadow Simlulator โ run real applications over a simulated Internet topology
We've started looking into eBPF a bit - IIUC eBPF by itself doesn't give us the ability to service or arbitrarily manipulate the traced process's syscalls.
We have recently learned of an interesting technique that dettrace [1] uses of combining seccomp with an eBPF filter and ptrace. Instead of generating a ptrace-stop for every syscall (as we do now, using PTRACE_SYSEMU), they use a seccomp policy with an eBPF filter, s.t. a ptrace-stop is only generated for syscalls that violate the policy, allowing them to emulate the result of those syscalls. syscalls that don't violate the policy are allowed to execute natively, saving a lot of overhead.
[1]: https://github.com/dettrace/dettrace
This works great for them since they want to emulate a relatively small subset of syscalls. In our case we want to emulate most syscalls, so it's not as clear-cut of a win. We have found though that if we use an LD_PRELOAD'd shim in the target process to intercept syscalls and then service them via IPC, that's substantially faster than catching them with ptrace. That runs back into the problems with LD_PRELOAD in general of there being various ways of missing syscalls. but, we may be able to use that technique along with ptrace+seccomp+ebpf to intercept any syscalls that we'd otherwise miss. The seccomp technique would allow us to exempt the syscalls that our shim itself is making to do the IPC.
shadow
-
Turmoil, a framework for developing and testing distributed systems
Cool, will be interested to see how this develops! tokio's loom framework has been a big help in testing some tricky concurrency code I've worked on.
Folks interested in this space might also be interested in the system I spend most of my time working on: Shadow. It also performs deterministic simulation of a network of hosts, but it intercepts network and system interactions at the syscall level via seccomp. As such it can work with binaries compiled from ~any language, usually without any code modification or special compilation. https://shadow.github.io/
-
I reinvented another wheel, linux threads.
Nice writeup! I've also had to dig a bit into this area in my work on shadow.
-
Shadow Simulation Developer
It is no longer active. If you are asking about Shadow, check out https://shadow.github.io
-
How to avoid bounds checks in Rust (without unsafe!)
I do share this hesitation. I think for simple cases iterators are usually fine, but I've definitely run into cases where an iterator adapter caused unexpected performance problems. e.g. https://github.com/shadow/shadow/pull/2543
-
Sending signals to Unix process groups
Yes. Though I'm not sure I see the connection to the OP...?
The example I'm most familiar with, because I work on it, is Shadow. We used ptrace for a bit but now use seccomp.
https://github.com/shadow/shadow/
- Shadow Simulator โ run real applications over a simulated Internet topology
-
Shadow Simlulator โ run real applications over a simulated Internet topology
For anyone interested in following current development on Shadow, we've been publishing a series of updates. Most recent: https://github.com/shadow/shadow/discussions/1274
The previous update has links back to the whole series; I stopped including it in the most-recent update since it was getting a bit cumbersome: https://github.com/shadow/shadow/discussions/1060
What are some alternatives?
mininet - Emulator for rapid prototyping of Software Defined Networks
core - Common Open Research Emulator
tor - unofficial git repo -- report bugs/issues/pull requests on https://gitlab.torproject.org/ --
shadow-plugin-tor - A Shadow plug-in that runs the Tor anonymity software
imunes - Integrated Multiprotocol Network Emulator/Simulator
rebop - Fast stochastic simulator for chemical reaction networks
reverie - An ergonomic and safe syscall interception framework for Linux.
testground - ๐งช A platform for testing, benchmarking, and simulating distributed and p2p systems at scale.