testy VS LazySmallCheck2012

Agreed with most of this but I’m skeptical of the rsc.io/script dsl approach. I’ll try it, though, because Russ is often right.

shameless advert: do you wish testify was implemented with generics and go-cmp, and had a more understandable surface area? Check out my small zero-dep library, Testy https://github.com/peterldowns/testy

shameless advert: do you want to write tests against your postgres database, but each new test adds seconds to your test suite? Check out pgtestdb, the marginal cost of each test is measured in tens of milliseconds, and each test gets a unique and isolated postgres instance — with all your migrations applied. https://github.com/peterldowns/pgtestdb

Finished, take a look if you're curious --

https://github.com/peterldowns/testy

Using laziness to avoid generating parts of an expression until it's needed is a really nice idea. The LazySmallCheck package[1] took this approach, but was limited in the types of data it could produce (e.g. it couldn't generate functions). This was extended by LazySmallCheck2012[2], but that seems to be unmaintained and doesn't work with more recent GHC versions.

(Note that these are named in reference to SmallCheck[3], which takes the approach of enumerating concrete values in order of "size"; as an alternative to the more widely-used QuickCheck[4], which generates concrete values at random, and tries to "shrink" those which trigger a failure)

[1] https://hackage.haskell.org/package/lazysmallcheck

[2] https://github.com/UoYCS-plasma/LazySmallCheck2012

[3] https://hackage.haskell.org/package/smallcheck

[4] https://hackage.haskell.org/package/QuickCheck

I tend to use anything that offers property-testing, since tests are much shorter to write and uncover lots more hidden assumptions.

My go-to choices per language are:

- Python: Hypothesis https://hypothesis.readthedocs.io/en/latest (also compatible with PyTest)

- Scala: ScalaCheck https://scalacheck.org (also compatible with ScalaTest)

- Javascript/Typescript: JSVerify https://jsverify.github.io

- Haskell: LazySmallCheck2012 https://github.com/UoYCS-plasma/LazySmallCheck2012/blob/mast...

- When I wrote PHP (over a decade ago) there was no decent property-based test framework, so I cobbled one together https://github.com/Warbo/php-easycheck

All of the above use the same basic setup: tests can make universally-quantified statements (e.g. "for all (x: Int), foo(x) == foo(foo(x))"), then the framework checks that statement for a bunch of different inputs.

Most property-checking frameworks generate data randomly (with more or less sophistication). The Haskell ecosystem is more interesting:

- QuickCheck was one of the first property-testing frameworks, using random genrators.

- SmallCheck came later, which enumerates data instead (e.g. testing a Float might use 0, 1, -1, 2, -2, 0.5, -0.5, etc.). That's cute, but QuickCheck tends to exercise more code paths with each input.

- LazySmallCheck builds up test data on-demand, using Haskell's pervasive laziness. Tests are run with an error as input: if they pass, we're done; if they fail, we're done; if they trigger the error, they're run again with slightly more-defined inputs. For example, if the input is supposed to be a list, we try again with the two forms of list: empty and "cons" (the arguments to cons are both errors, to begin with). This exercises even more code paths for each input.

- LazySmallCheck2012 is a more versatile "update" to LazySmallCheck; in particular, it's able to generate functions.