smallcheck VS LazySmallCheck2012

Using laziness to avoid generating parts of an expression until it's needed is a really nice idea. The LazySmallCheck package[1] took this approach, but was limited in the types of data it could produce (e.g. it couldn't generate functions). This was extended by LazySmallCheck2012[2], but that seems to be unmaintained and doesn't work with more recent GHC versions.

(Note that these are named in reference to SmallCheck[3], which takes the approach of enumerating concrete values in order of "size"; as an alternative to the more widely-used QuickCheck[4], which generates concrete values at random, and tries to "shrink" those which trigger a failure)

[1] https://hackage.haskell.org/package/lazysmallcheck

[2] https://github.com/UoYCS-plasma/LazySmallCheck2012

[3] https://hackage.haskell.org/package/smallcheck

[4] https://hackage.haskell.org/package/QuickCheck

Could you compare with https://hackage.haskell.org/package/smallcheck in particular the smallcheck ability "to verify properties for all test cases up to some depth"?

Another strategy is exhaustive generation. There, all possible values for some type are generated in some well-defined order - typically from "small" to "large" values, and with some upper bound, as once you go past booleans the number of values for most types are (countably) infinite. For example, trying all the integers between -20 and 20 in "zig zag" order 0,1,-1,2,-2,.... SmallCheck for Haskell and SciFe for Scala do this, but this approach is not so well-known. It's a shame as random and exhaustive generation are complementary - if you think of generating values as exploring some large space to find failing tests, random generation is a serendipitous type of exploration, while exhaustive generation is diligently mapping out all the paths in some area.

Using laziness to avoid generating parts of an expression until it's needed is a really nice idea. The LazySmallCheck package[1] took this approach, but was limited in the types of data it could produce (e.g. it couldn't generate functions). This was extended by LazySmallCheck2012[2], but that seems to be unmaintained and doesn't work with more recent GHC versions.

(Note that these are named in reference to SmallCheck[3], which takes the approach of enumerating concrete values in order of "size"; as an alternative to the more widely-used QuickCheck[4], which generates concrete values at random, and tries to "shrink" those which trigger a failure)

[1] https://hackage.haskell.org/package/lazysmallcheck

[2] https://github.com/UoYCS-plasma/LazySmallCheck2012

[3] https://hackage.haskell.org/package/smallcheck

[4] https://hackage.haskell.org/package/QuickCheck

I tend to use anything that offers property-testing, since tests are much shorter to write and uncover lots more hidden assumptions.

My go-to choices per language are:

- Python: Hypothesis https://hypothesis.readthedocs.io/en/latest (also compatible with PyTest)

- Scala: ScalaCheck https://scalacheck.org (also compatible with ScalaTest)

- Javascript/Typescript: JSVerify https://jsverify.github.io

- Haskell: LazySmallCheck2012 https://github.com/UoYCS-plasma/LazySmallCheck2012/blob/mast...

- When I wrote PHP (over a decade ago) there was no decent property-based test framework, so I cobbled one together https://github.com/Warbo/php-easycheck

All of the above use the same basic setup: tests can make universally-quantified statements (e.g. "for all (x: Int), foo(x) == foo(foo(x))"), then the framework checks that statement for a bunch of different inputs.

Most property-checking frameworks generate data randomly (with more or less sophistication). The Haskell ecosystem is more interesting:

- QuickCheck was one of the first property-testing frameworks, using random genrators.

- SmallCheck came later, which enumerates data instead (e.g. testing a Float might use 0, 1, -1, 2, -2, 0.5, -0.5, etc.). That's cute, but QuickCheck tends to exercise more code paths with each input.

- LazySmallCheck builds up test data on-demand, using Haskell's pervasive laziness. Tests are run with an error as input: if they pass, we're done; if they fail, we're done; if they trigger the error, they're run again with slightly more-defined inputs. For example, if the input is supposed to be a list, we try again with the two forms of list: empty and "cons" (the arguments to cons are both errors, to begin with). This exercises even more code paths for each input.

- LazySmallCheck2012 is a more versatile "update" to LazySmallCheck; in particular, it's able to generate functions.