go-fuzz VS trophy-case

I used this method successfully for my qjson package . It accepts as input a human readable json. It detected a condition I forgot to check in a few minutes. I used the go fuzzer go-fuzz from Dmitry Vyukov. Check the impressive list of trophies at the end of the README. These are bugs found by the fuzzer.

It does instrumented fuzzing. The older https://github.com/dvyukov/go-fuzz would rewrite your sources to inject the instrumentation and pass the rewritten sources to the compiler, but it didn't really work with Go modules. This is something that probably makes sense to integrate with the compiler toolchain, same as `go test`'s coverage testing.

And another Go-Fuzz

Anyone seen good articles on converting go-fuzz tests to native fuzzing? Specifics on the new corpus format and a converter from go-fuzz would be really useful.

It’s great to hear that the fuzzer is built on go-fuzz so hopefully the conversion process won’t be too bad: https://github.com/dvyukov/go-fuzz/issues/329

> it shows the limitations of the language that you can't just build this inside the language.

Not sure why you'd make that assumption. https://github.com/dvyukov/go-fuzz

Rather than explore the specific issues my tests found, I started playing with running https://github.com/dvyukov/go-fuzz against go-json which seems to have fairly quickly found some issues. I've opened https://github.com/goccy/go-json/issues/174 to start the conversation about how you want to proceed here.

You can check the go-fuzz's trophies here.

Maybe you can add support for using a fuzzer like https://github.com/dvyukov/go-fuzz or https://github.com/google/gofuzz

I recently stumbled upon go-fuzz (that is also linked in the proposal). I'm amazed how fuzzing has made the go toolchain stronger and integrated this technique into some of our CI/CD pipelines for continuous testing. Having native fuzzy testing support (not just in go) would be great.

Go tests and benchmarks are so easy to write and run: just add TestFoo and BenchmarkFoo functions to a bar_test.go file, and "go test" does the rest. It's currently doable, but it requires a 3rd party library (go-fuzz) and a bit of fluffing around. This will make fuzz testing an equally first-class citizen with standard Go tooling (just add FuzzFoo), and as such we'll probably see a lot more people testing with fuzzing.

I used go-fuzz in GoAWK and it found several bugs (see https://benhoyt.com/writings/goawk/#fuzz-testing), and almost everyone who's done fuzz testing has similar reports. Certainly go-fuzz has found many, many bugs in Go itself: https://github.com/dvyukov/go-fuzz#trophies

For what it's worth, I wrote an article for LWN about the upcoming support for built-in fuzzing in Go: https://lwn.net/Articles/829242/ (of course, if you want full details, read the full proposal).

You could check cargo-fuzz trophy case, which is a list of issues that have been found via fuzzing.

I've added it to the trophy case.

That said, what's present in what quantities under what circumstances in the Rust fuzzing trophy case does a pretty good job of illustrating how effective the Rust compiler is at ruling out entire classes of bugs.

The same fuzzing techniques applied to Rust yielded a lot of bugs as well. But in Rust's case only 7 out of 340 fuzzer-discovered bugs, or 2%, were memory corruption issues. Naturally, all of the memory corruption bugs were in unsafe code.

https://github.com/rust-fuzz/trophy-case has like 70 of my issues in it, including the nine minidump bugs

If you have found any bugs with this tool, perhaps add them to the Rust fuzz trophy case?

Source: https://github.com/rust-fuzz/trophy-case (over 40 of those are just from me).

But to bring some data, check out the fuzz trophy case. It shows that failures in Rust are most often assertions/panics (equivalent to C++ exception) with memory corruption being relatively rare (it's not never—Rust isn't promising magic—but it's a significant change).

You need to read the list more carefully.

• The list is not for Rust itself, but every program every written in Rust. By itself it doesn't mean much, unless you compare prevalence of issues among Rust programs to prevalence of issues among C programs. For some context, see how memory unsafety is rare compared to assertions and uncaught exceptions: https://github.com/rust-fuzz/trophy-case

• Many of the memory-unsafety issues are on the C FFI boundary, which is unsafe due to C lacking expressiveness about memory ownership of its APIs (i.e. it shows how dangerous is to program where you don't have the Rust borrow checker checking your code).

• Many bugs about missing Send/Sync or evil trait implementations are about type-system loopholes that prevented compiler from catching code that was already buggy. C doesn't have these guarantees in the first place, so lack of them is not a CVE for C, but just how C is designed.