test-framework by Max Bolingbroke

A parallel test framework for combining tests made using QuickCheck and HUnit and much more...

Description

A common testing framework that allows tests such as QuickCheck properties and HUnit test cases to used together easily. Furthermore, it enhances both styles of test with all of these great features:

Run tests in parallel but report results in a deterministic order (to aid diff-based analysis of test output).
Progress reporting for individual QuickCheck properties and for the whole suite being run.
Filter the tests to be run using patterns specified on the command line.
Hierarchical, colored display of test results.
Reporting of test statistics (number of tests run, number failed etc.).
Extensibility: add your own test providers above and beyond those provided.
Reporting of the seed used upon a failed QuickCheck run, so you can reproduce the failure if necessary.

If you like test-framework you may also want to check out Oscar Finnsson's test-framework-th package for generating your test groups programatically.

Example

The latest version of the example is always available online:

We can run these tests from the command line (in parallel!) like so:

$ ghc -package test-framework -package test-framework-quickcheck \
    -package test-framework-hunit -threaded Example.hs -o Example
$ ./Example --maximum-generated-tests=5000 +RTS -N2

FAQ

I tried to run the example, and got an error about not having a Testable instance

You may get an error like the following:

No instance for (QuickCheck-1.2.0.0:Test.QuickCheck.Testable
                      (Gen Prop))
     arising from a use of `testProperty' at Example.lhs:68:16-46

The reason is that GHC is building the example (which uses QuickCheck 1) with the QuickCheck 2 library. You can fix this by running:

$ ghc-pkg hide QuickCheck-2.1.0.1

You may need adjust the version number to match the one on your machine, as reported by ghc-pkg list. Alternatively, you can supply the -package QuickCheck-1.2.0.0 flag when invoking GHC (again you may need to change the version number to match your installed QuickCheck version).

I have some tests that just call `error` but the tests pass

This happens because GHC finds it convenient to eta-expand the calls to error, so we can no longer observe the failure. This is dodgy behaviour by GHC, but it's unlikely to change any time soon, so the workaround is to compile your tests with -O0 (the default).

If you really do want to compile your tests with optimisations enabled, you should also be able to supply the -fno-full-laziness flag along with your -O1 or -O2 flag. However, if new optimisations are added to GHC they may break this workaround - it's probably safest to turn optimisation off altogether.

Providers

The framework is built with an extensible architecture that allows anyone to add their own test "provider". All of the supplied providers are built using this mechanism, and so you probably want to install not only the infrastructure package (test-framework), but also the commonly-used providers test-framework-quickcheck and test-framework-hunit.

The Console Test Runner

A description of the options available can be obtained by using the option --help on the command line.

The test-selection syntax for use with the -s command line option is based on that of shell globs or Git .gitignore files. Test patterns consist of the following:

An optional prefixed bang ! which negates the pattern.
If the pattern ends with a slash, it is removed for the purpose of the following description, but it would only find a match with a test group. In other words, foo/ will match a group called foo and any tests underneath it, but will not match a regular test foo.
If the pattern does not contain a slash /, the framework checks for a match against any single component of the path.
Otherwise, the pattern is treated as a glob, where:
- The wildcard * matches anything within a single path component (i.e. foo but not foo/bar).
- Two wildcards ** matches anything (i.e. foo and foo/bar).
- Anything else matches exactly that text in the path (i.e. foo would only match a component of the test path called foo (or a substring of that form).

For example, group/*1 matches group/test1 but not group/subgroup/test1, whereas both examples would be matched by group/**1. A leading slash matches the beginning of the test path; for example, /test* matches test1 but not group/test1.

A test will be run if it matches any of the patterns supplied with -s.

Documentation

Haddock documentation is available at Hackage:

test-framework
test-framework-quickcheck
test-framework-quickcheck2
test-framework-hunit

Download

Install with Cabal:

$ cabal install test-framework
$ cabal install test-framework-quickcheck
$ cabal install test-framework-quickcheck2
$ cabal install test-framework-hunit

Download this project in either zip or tar formats.

Clone the project with Git:

$ git clone git://github.com/batterseapower/test-framework

Alternatively, browse the source on GitHub

Contact

You can submit a ticket at Lighthouse to report a bug.

Other enquiries can be directed to me via email