JQF VS fuzzcheck-rs

We are working on projects related with cybersecurity and compilers. A reference we look at is [1] and [2]. I think we can publish the results in the coming months.

[1] https://github.com/rohanpadhye/jqf/wiki/Fuzzing-a-Compiler

[2] https://news.ycombinator.com/item?id=36373410

QuickcCheck-type tools (generators for tests that know about the edge cases of a domain - e. g. for the domain of numbers considering things like 0, the infinities, various almost-and-just-over powers of two, NaN and mantissas for floats, etc.):

* QuickCheck: https://hackage.haskell.org/package/QuickCheck

* Hypothesis: https://hypothesis.readthedocs.io/en/latest/

* JUnit QuickCheck: https://github.com/pholser/junit-quickcheck

Fuzz testing tools (tools which mutate the inputs to a program in order to find interesting / failing states in that program). Generally paired with code coverage:

* American Fuzzy Lop (AFL): https://github.com/google/AFL

* JQF: https://github.com/rohanpadhye/JQF

Mutation / Fault based test tools (review your existing unit coverage and try to introduce changes to your _production_ code that none of your tests catch)

* PITest: https://pitest.org/

If you are interested in fuzzing your Java code, you should also have a look at the JQF project which directly integrates with junit tests: https://github.com/rohanpadhye/JQF

Fuzzcheck is a structure-aware fuzzer for rust. "Fuzzing" means feeding large amounts of data into a program and checking for crashes (Fuzzcheck also checks to make sure that all the properties your program should uphold – e.g. a sorting algorithm applied to a list of n items should always return a list of n items – are upheld). Fuzzcheck is an "evolutionary" fuzzer – this means that it generates a set of random inputs, sees what percentage of the program is executed for each input, and keeps inputs which have high levels of percentage of program executed. It then "mutates" these inputs – whereas fuzzers such as AFL/Hongfuzz/etc mutate raw bytes in place (e.g. they swap bytes at different positions, or insert a random byte at a given position to generate inputs similar to the chosen "high coverage" inputs), Fuzzcheck works directly on the Rust types (so it might swap the order of two items in a vec, or randomly insert a new item). It's a really powerful tool for finding lots of bugs.

If you want help with Win support (issues/8) maybe post it here to get it added to TWIR.

I am working on a code coverage viewer for my fuzzer (fuzzcheck). I described what I've done so far in this issue and I am hoping to release the first version within two weeks.

The implications for my fuzzer, fuzzcheck, are huge! Compiling fuzz tests is a lot easier. There should be no more need to create a separate fuzz folder, fuzz tests can be regular #[test] functions, private implementation details can be fuzz-tested as well, rust-analyser works as expected, documentation can be easily generated, etc. I can also attach a human-readable coverage report to every test case :)

Since I graduated, I have had a lot more time to work on fuzzcheck. I am trying to flesh it out, test it, and document it for a new release. It has always felt a bit rushed/experimental and now I am hoping to make it into something solid. I have also played with an egui interface for it, to visualise the tested code coverage, understand how the fuzzer’s decisions are made, and also to interactively tweak the fuzzer’s behaviour. It's a lot of work but it's slowly all coming together! :)

fuzzcheck-rs is really cool. It combines property-testing with fuzzing, getting the nice, structured nature of the former, and the coverage-driven search of the latter, but it works by mutating the structure directly instead of going through a bit string. So if you have a binary tree, going from A(B, C) to A(C, B) can be a single mutation away if that makes sense in your use case, instead of being arbitrarily far away in the bitstring approach.