[petsc-dev] Introducing new "test harness" to PETSc

[Patrick Sanan]

This is really great!  If they don't already do so, maybe it would be
simple now to modify the new example-walking scripts to ensure that
the test suite covers all the examples/tutorials (maybe even nightly),
noting any qualifying files that don't have a valid /*TEST .. TEST*/
block.

On Fri, Jan 13, 2017 at 11:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>    PETSc developers,
>
>      Scott Kruger has been implementing a new "test harness" for PETSc. Rather than having tests implemented as bash lines in the makefiles, the tests will be defined within the source code for each example with a simple format and the actual scripts that run the tests will be automatically generated. This offers three advantages over the current system
>
> 1) test definitions can be given in a very concise format that is independent of how the test is run
> 2) multiple approaches to actually running the tests can be implemented without requiring any changes to the test definitions for each example.
> 3) the tests can be compiled and run in parallel (eventually we hope to see much faster times for testing).
>
>    Here is an example of the format that defines tests
>
>      test:
>
>    test:
>       suffix: yaml
>       requires: yaml !complex
>       args: -options_file_yaml bag.yml -options_view
>       filter:  grep -v saws_port_auto_select
>       localrunfiles: bag.yml
>
> The first is a test that takes no command line arguments and does no filtering of output. The second takes command line arguments, does some filtering of the output and this example also requires a data file associated with that example.  The expected output of tests continues to reside in the output directory with the previously naming convention of XXX_1.out XXX_2.out where XXX is the name of the example or XXX_suffix.out where suffix is defined in the test definition.
>
>   So far in the master branch we have only converted over two directories of examples sys/examples/tutorials and sys/examples/tests so you can look in those directories for other examples. Over the next few weeks we hope to convert over all the rest of the directories.
>
>   Special thanks to Scott, Satish, and Jed for all the work they have done to set up the new test harness.
>
>    Barry
>
>    Technical details: the new python script ./config/gmakegentest.py which is normally automatically run at configure walks through the examples directories looking for the formatted test definitions, for each test it generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or tutorials/ a shell script that can run that test. In addition config/gmakegentest.py  adds dependencies to the $PETSC_DIR/gmakefile for all the tests that it has found. The rule make alltests then executes all those dependencies.
>
>    We will be refining both the testing harness and the test definitions as we convert the rest of the examples over to the new approach.
>
>
>
>
>