[petsc-dev] Introducing new "test harness" to PETSc

[Barry Smith]

   PETSc developers,

     Scott Kruger has been implementing a new "test harness" for PETSc. Rather than having tests implemented as bash lines in the makefiles, the tests will be defined within the source code for each example with a simple format and the actual scripts that run the tests will be automatically generated. This offers three advantages over the current system

1) test definitions can be given in a very concise format that is independent of how the test is run
2) multiple approaches to actually running the tests can be implemented without requiring any changes to the test definitions for each example.
3) the tests can be compiled and run in parallel (eventually we hope to see much faster times for testing).

   Here is an example of the format that defines tests

     test:

   test:
      suffix: yaml
      requires: yaml !complex
      args: -options_file_yaml bag.yml -options_view
      filter:  grep -v saws_port_auto_select
      localrunfiles: bag.yml

The first is a test that takes no command line arguments and does no filtering of output. The second takes command line arguments, does some filtering of the output and this example also requires a data file associated with that example.  The expected output of tests continues to reside in the output directory with the previously naming convention of XXX_1.out XXX_2.out where XXX is the name of the example or XXX_suffix.out where suffix is defined in the test definition.

  So far in the master branch we have only converted over two directories of examples sys/examples/tutorials and sys/examples/tests so you can look in those directories for other examples. Over the next few weeks we hope to convert over all the rest of the directories. 

  Special thanks to Scott, Satish, and Jed for all the work they have done to set up the new test harness.

   Barry

   Technical details: the new python script ./config/gmakegentest.py which is normally automatically run at configure walks through the examples directories looking for the formatted test definitions, for each test it generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or tutorials/ a shell script that can run that test. In addition config/gmakegentest.py  adds dependencies to the $PETSC_DIR/gmakefile for all the tests that it has found. The rule make alltests then executes all those dependencies. 

   We will be refining both the testing harness and the test definitions as we convert the rest of the examples over to the new approach.