[petsc-dev] test suite questions

[Dominic Meiser]

Hi,

I have a few questions regarding usage of the PETSc test suite. Some of 
the questions are related to issues discussed in this thread 
https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2013-January/010765.html.

1) Currently I manually set the DATAFILESPATH environment variable. Is 
this how I'm supposed to do this or is this handled differently by the 
test suite?

2) In the above thread the issue of size of the test suite was brought 
up with the time it takes to run `make alltest` being a concern. While 
working on bug fixes my workflow tends to be: Find a test case that 
demonstrates the bug (usually run an existing test with different 
command line options); fix the bug; move on to next related bug. This 
can produce several test cases that sometimes overlap in substantial 
ways. For instance a bug might be present in serial and parallel. With 
my workflow I would end up with a serial and a parallel test. While this 
has the advantage of making it easier to narrow down a failure it has 
the disadvantage of `make alltest` taking longer. How should I balance 
these two aspects?

3) What is your workflow for examining test results? Do you scan test 
results manually? Or is it mostly sufficient to only pay attention to 
build failures and crashes?

4) Clearly it would be desirable to have tests where we can 
automatically (without manual inspection) decide if the test passes or 
fails. This was also discussed in the above thread. Perhaps it should be 
a requirement that new tests are written in such a way that failure 
detection can be done automatically? Typically these tests will then be 
less suitable as examples for getting started and they should go into 
the test rather than tutorial directories. Clearly scientific computing 
has additional challenges compared to unit testing in "normal" software 
engineering when it comes to automatically detecting test failure. Some 
of the techniques we use at Tech-X include:
- hdfdiff (similar to exodiff)
- ndiff (diffing with atol and rtol for plain text)
- Decide in the driver program whether a test has passed or failed.
The last option is the most flexible. For instance one could have logic 
like the following (very schematically):
if (numIters > 10 && numIters < 15 && residual < 1.0e-9) {
     printf("SUCCESS\n");
} else {
     printf("FAILURE\n");
}
Combined with a disciplined approach for the output of a test it's 
possible to write strong tests with a much lower likelihood of false 
positives. If a quantity is overly sensitive to valid code or 
configuration modifications then don't print it. Obviously this is much 
easier to do if a test does not have the dual purpose of a tutorial.

Thanks,
Dominic

-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com