Ansi-test Alternatives
Similar projects and alternatives to ansi-test
-
Schemathesis
Automate your API Testing: catch crashes, validate specs, and save time
-
hypothesis
Hypothesis is a powerful, flexible, and easy to use library for property-based testing.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
ansi-test reviews and mentions
- ECL could run as fast as SBCL if it does more type inference and aggressive function inlining
-
What Is Property Based Testing?
I have applied it to testing Common Lisp implementations, but there has been much work on applying it to other languages (most famously C and Javascript.)
I applied the following techniques:
(1) Generate random valid well defined programs and see (a) if they crash the compiler, (b) cause different CL implementations to produce different outputs, (c) when modified (by addition of randomly generated optimization directives or valid type declarations) they still produce the same output. This is differential testing, which was used by McKeeman at DEC in the 1990s to test C compilers, and later improved (again, on C compilers) by Yang, Chan, Eide, and Regehr at U. of Utah (csmith, creduce).
Since a running lisp image can generate and compile functions internally (this IS lisp, after all), the testing loop can be very fast. Since 2003 I have run this on and off for many billions of iterations on desktop and laptop machines on various CL implementations, now mainly on SBCL. Most of the test input reduction is handled automatically, which is a big help.
(2) Generate random possibly invalid code by mutating or recombining snippets drawn from a large corpus of code, to see if it crashes the compiler (in CL implementations where the compiler is promised to never respond to bad code by signaling an error.) This was also the approach jsfunfuzz took on Javascript.
(3) Extensive fuzzing of calls to standard functions in CL, using random generation of input values and random generation of valid type declarations, with the invariant being that the same values should be computed (and the compiler not fail.) This is a specialization of (1), but was sufficiently different that the bugs it found were not the same.
Examples of tests produced by (1) and (3) over an early period when this was being developed. Each caused a failure in some CL implementation: https://github.com/pfdietz/ansi-test/blob/master/misc/misc.l...
See also the various bugs I've reported against SBCL over the years, many of which come from this testing. https://bugs.launchpad.net/~paul-f-dietz
The experience with this sort of testing of compilers (in any language) is that if the compiler (free or commercial) has never been subjected to it, it will immediately find bugs in the compiler.
Stats
pfdietz/ansi-test is an open source project licensed under MIT License which is an OSI approved license.
The primary programming language of ansi-test is Common Lisp.
