fuzzcheck-rs VS trophy-case

Fuzzcheck is a structure-aware fuzzer for rust. "Fuzzing" means feeding large amounts of data into a program and checking for crashes (Fuzzcheck also checks to make sure that all the properties your program should uphold – e.g. a sorting algorithm applied to a list of n items should always return a list of n items – are upheld). Fuzzcheck is an "evolutionary" fuzzer – this means that it generates a set of random inputs, sees what percentage of the program is executed for each input, and keeps inputs which have high levels of percentage of program executed. It then "mutates" these inputs – whereas fuzzers such as AFL/Hongfuzz/etc mutate raw bytes in place (e.g. they swap bytes at different positions, or insert a random byte at a given position to generate inputs similar to the chosen "high coverage" inputs), Fuzzcheck works directly on the Rust types (so it might swap the order of two items in a vec, or randomly insert a new item). It's a really powerful tool for finding lots of bugs.

If you want help with Win support (issues/8) maybe post it here to get it added to TWIR.

I am working on a code coverage viewer for my fuzzer (fuzzcheck). I described what I've done so far in this issue and I am hoping to release the first version within two weeks.

The implications for my fuzzer, fuzzcheck, are huge! Compiling fuzz tests is a lot easier. There should be no more need to create a separate fuzz folder, fuzz tests can be regular #[test] functions, private implementation details can be fuzz-tested as well, rust-analyser works as expected, documentation can be easily generated, etc. I can also attach a human-readable coverage report to every test case :)

Since I graduated, I have had a lot more time to work on fuzzcheck. I am trying to flesh it out, test it, and document it for a new release. It has always felt a bit rushed/experimental and now I am hoping to make it into something solid. I have also played with an egui interface for it, to visualise the tested code coverage, understand how the fuzzer’s decisions are made, and also to interactively tweak the fuzzer’s behaviour. It's a lot of work but it's slowly all coming together! :)

fuzzcheck-rs is really cool. It combines property-testing with fuzzing, getting the nice, structured nature of the former, and the coverage-driven search of the latter, but it works by mutating the structure directly instead of going through a bit string. So if you have a binary tree, going from A(B, C) to A(C, B) can be a single mutation away if that makes sense in your use case, instead of being arbitrarily far away in the bitstring approach.

You could check cargo-fuzz trophy case, which is a list of issues that have been found via fuzzing.

I've added it to the trophy case.

That said, what's present in what quantities under what circumstances in the Rust fuzzing trophy case does a pretty good job of illustrating how effective the Rust compiler is at ruling out entire classes of bugs.

The same fuzzing techniques applied to Rust yielded a lot of bugs as well. But in Rust's case only 7 out of 340 fuzzer-discovered bugs, or 2%, were memory corruption issues. Naturally, all of the memory corruption bugs were in unsafe code.

https://github.com/rust-fuzz/trophy-case has like 70 of my issues in it, including the nine minidump bugs

If you have found any bugs with this tool, perhaps add them to the Rust fuzz trophy case?

Source: https://github.com/rust-fuzz/trophy-case (over 40 of those are just from me).

But to bring some data, check out the fuzz trophy case. It shows that failures in Rust are most often assertions/panics (equivalent to C++ exception) with memory corruption being relatively rare (it's not never—Rust isn't promising magic—but it's a significant change).

You need to read the list more carefully.

• The list is not for Rust itself, but every program every written in Rust. By itself it doesn't mean much, unless you compare prevalence of issues among Rust programs to prevalence of issues among C programs. For some context, see how memory unsafety is rare compared to assertions and uncaught exceptions: https://github.com/rust-fuzz/trophy-case

• Many of the memory-unsafety issues are on the C FFI boundary, which is unsafe due to C lacking expressiveness about memory ownership of its APIs (i.e. it shows how dangerous is to program where you don't have the Rust borrow checker checking your code).

• Many bugs about missing Send/Sync or evil trait implementations are about type-system loopholes that prevented compiler from catching code that was already buggy. C doesn't have these guarantees in the first place, so lack of them is not a CVE for C, but just how C is designed.