Stryker.NET VS FsCheck

Bogus - For creating fake data Verify - Snapshot testing for .NET MELT - For testing ILogger usage Stryker - Mutation Testing for .NET TestContainers - run docker programmatically in integration tests

This is called mutation testing btw.

Let's try it on our small example using Stryker.

There are tools that do this automatically, stryker[2] is one of them. When you run stryker, it will create many mutant versions of your production code, and run your tests for each mutant (that's how mutations are called in stryker's' documentation) version of the code. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. Let's have a look at the the result of runnning stryker against reffects-store's code:

Mutation testing is pretty solid. Better than code coverage for sure. Using Stryker personally.

A mutation testing tool (like Stryker) runs your unit tests to verify they all pass then makes a small change (mutation) to your code and reruns the tests. At least one test should fail because the modified code should behave differently.

I haven't tried it yet, but last time I researched it, this is the library that looked most promising: https://stryker-mutator.io/

Website: https://stryker-mutator.io/

As you can see from the imports statement we're relying on FsCheck to generate some random values for us.

C# - FsCheck

https://fscheck.github.io/FsCheck/ can hopefully generate random inputs automatically or with low effort for many methods to get your code coverage up. You don’t even need to write real tests right now, just call the methods with the random inputs and check they don’t fail.

Try using F# for tests. It has some great libraries like FsCheck (https://fscheck.github.io/FsCheck/).

At Symbolica we're building a symbolic execution service that explores every reachable state of a user's program and verifies assertions at each of these states to check that the program is correct. By default it will check for common undefined behaviours, such as out-of-bounds memory reads or divide by zero, but it can also be used with custom, application specific, assertions too just like the kind you'd write in a unit test. Seen from this perspective it's kind of like FsCheck (or Haskell's QuickCheck or Python's Hypothesis), but much more exhaustive and without the randomness.

Maybe something like QuickCheck, a quick search gave me this library for .NET https://github.com/fscheck/FsCheck

Because of the above I've generally been using tools like Stryker.NET and FsCheck to augment my testing suite. I'm still doing unit testing to find the more obvious "I haven't had my coffee, let's make sure I'm doing what I think I'm doing" bugs. I'm just using things like mutation testing, property testing, fuzzing, etc. to find the deeper issues in my code. There's a ton of libraries out there, including one that I've built for myself to help with testing but FsCheck and Stryker are just beautiful. And if you're interested in fuzzing, SharpFuzz is a great option. But that one isn't quite as easy of an on ramp compared to the other two that I mentioned.

Looks cool. Is there a reason why you didn't use FsCheck or Hedgehog? They're built to generate random data for testing, and can return the seed if a test fails so you can rerun the test with the exact same data once you figure out what the problem is - which is useful if the failure condition is rare.

Haskell has QuickCheck and Hedgehog, and dotnet has both as well. F# is favored, but there's C# interop.

Side note, if you are thinking about testing in general, might be interested in property based testing. See for example https://fscheck.github.io/FsCheck/