clusterfuzz VS serenity

https://github.com/google/clusterfuzz

At least Chromium has integrated multiple different fuzzers into their regular development workflow and found lots of bugs even before going public.

Then it is clear that the behavior of this for loop is either not important or not being tested. This could mean that the tests that you do have are not useful and can be deleted.

> For most non-trivial software the possible state-space is enormous and we generally don't/can't test all of it. So "not testing the (full) behaviour of your application is the default for any test strategy", if we could we wouldn't have bugs... Last I checked most software (including Google's) has plenty of bugs.

I have also used (setup, fixed findings) using https://google.github.io/clusterfuzz/ which uses coverage + properties to find bugs in the way C++ code handles pointers and other things.

> The next question would be let's say I spend my time writing the tests to resolve this (could be a lot of work) is that time better spent vs. other things I could be doing? (i.e. what's the ROI)

That is something that will depend largely on the team and the code you are on. If you are in experimental code that isn't in production, is there value to this? Likely not. If you are writing code that if it fails to parse some data correctly you'll have a huge headache trying to fix it? Likely yes.

The SRE workbook goes over making these calculations.

> Even ignoring that is there data to support that the quality of software where mutation testing was added improved measurably (e.g. less bugs files against the deployed product, better uptime, etc?)

I know that there are studies that show that tests reduce bugs but I do not know of studies that say that higher test coverage reduces bugs.

The goal of mutation testing isn't to drive up coverage though. It is to find out what cases are not being exercised and evaluating if they will cause a problem. For example mutation testing tools have picked up cases like this:

   if (debug) print("Got here!");

SerenityOS replicates that look and feel. It is also implemented in a dialect of C++ that adheres to some of the good parts of C++98: https://serenityos.org

One example of a useful technique

https://serenityos.org/ apparently only makes source code available. There are no binary images of the OS to install

I think Andreas said this functions like a little test -- if you're not willing to build it from source, then you probably wouldn't be a good contributor anyway.

---

Likewise, my shell project provides source tarballs only, right now - https://www.oilshell.org/release/0.21.0/

It is packaged in a number of places, which I appreciate. That means some other people are willing to do some work.

And they provide good feedback.

I would like it to be more widely available, but yeah I definitely see that you need to "gate" peanut gallery feedback a bit, because it takes up a lot of time.

Of course, it's a tricky balance, because you also want feedback from casual users, to make the project better.

Indeed, given the existence of `JS::NonnullGCPtr`, `JS::GcPtr` intentionally corresponds to a nullable pointer, so it seems dangerous to convert one to a reference without a null-check.

That said, a naive code search finds what *may* be more cases of this pattern:

https://github.com/search?q=repo%3ASerenityOS%2Fserenity+%2F...

Eg: https://github.com/SerenityOS/serenity/blob/a68b134e6dea5065... -> https://github.com/SerenityOS/serenity/blob/a68b134e6dea5065...

In some of those search results, it is fine because there is a preceding null-check, and obviously I know nothing about this code other than this naive search result, but perhaps it would be prudent to vet all of them.

It is a SerenityOS project. You can find the answer to that question in their primary project's FAQ[1].

1. https://github.com/SerenityOS/serenity/blob/master/Documenta...

https://github.com/SerenityOS/serenity

The best way to write proper exception free C++ is not to use the C++ Standard Library.

I initially confused it with Serenity OS prior to watching the video: https://github.com/SerenityOS/serenity

My contributions to SerenityOS[0] helped me get my current job. My team lead (who was also my interviewer) was interested in what I did since I listed some of it in my CV, and I showed him some PRs I made and explained what went into each of them. It was really exciting because I didn't have professional experience with low-level development, and basically got the job due to hobby programming.

[0]: https://github.com/SerenityOS/serenity/pulls?q=is%3Apr+autho...

Definitely not "literally impossible", just a great deal of work. https://github.com/SerenityOS/serenity/tree/master/Ladybird