[petsc-dev] test suite questions

[Barry Smith]

On Jul 24, 2014, at 10:56 AM, Dominic Meiser <dmeiser at txcorp.com> wrote:

> Hi,
> 
> I have a few questions regarding usage of the PETSc test suite. Some of the questions are related to issues discussed in this thread https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2013-January/010765.html.
> 
> 1) Currently I manually set the DATAFILESPATH environment variable. Is this how I'm supposed to do this or is this handled differently by the test suite?

   Set it when you run configure then the test suite will run those tests. —DATAFILESPATH=  see datafilespath.py   it also looks for a directory next to the petsc root directory called datafiles  you can add any other reasonable locations to this file (for example ~/Datafiles)

  Barry


> 
> 2) In the above thread the issue of size of the test suite was brought up with the time it takes to run `make alltest` being a concern. While working on bug fixes my workflow tends to be: Find a test case that demonstrates the bug (usually run an existing test with different command line options); fix the bug; move on to next related bug. This can produce several test cases that sometimes overlap in substantial ways. For instance a bug might be present in serial and parallel. With my workflow I would end up with a serial and a parallel test. While this has the advantage of making it easier to narrow down a failure it has the disadvantage of `make alltest` taking longer. How should I balance these two aspects?
> 
> 3) What is your workflow for examining test results? Do you scan test results manually? Or is it mostly sufficient to only pay attention to build failures and crashes?
> 
> 4) Clearly it would be desirable to have tests where we can automatically (without manual inspection) decide if the test passes or fails. This was also discussed in the above thread. Perhaps it should be a requirement that new tests are written in such a way that failure detection can be done automatically? Typically these tests will then be less suitable as examples for getting started and they should go into the test rather than tutorial directories. Clearly scientific computing has additional challenges compared to unit testing in "normal" software engineering when it comes to automatically detecting test failure. Some of the techniques we use at Tech-X include:
> - hdfdiff (similar to exodiff)
> - ndiff (diffing with atol and rtol for plain text)
> - Decide in the driver program whether a test has passed or failed.
> The last option is the most flexible. For instance one could have logic like the following (very schematically):
> if (numIters > 10 && numIters < 15 && residual < 1.0e-9) {
>    printf("SUCCESS\n");
> } else {
>    printf("FAILURE\n");
> }
> Combined with a disciplined approach for the output of a test it's possible to write strong tests with a much lower likelihood of false positives. If a quantity is overly sensitive to valid code or configuration modifications then don't print it. Obviously this is much easier to do if a test does not have the dual purpose of a tutorial.
> 
> Thanks,
> Dominic
> 
> -- 
> Dominic Meiser
> Tech-X Corporation
> 5621 Arapahoe Avenue
> Boulder, CO 80303
> USA
> Telephone: 303-996-2036
> Fax: 303-448-7756
> www.txcorp.com
>