[petsc-dev] Introducing new "test harness" to PETSc

[Barry Smith]

> On Jan 13, 2017, at 4:44 PM, Patrick Sanan <patrick.sanan at gmail.com> wrote:
> 
> This is really great!  If they don't already do so, maybe it would be
> simple now to modify the new example-walking scripts to ensure that
> the test suite covers all the examples/tutorials (maybe even nightly),
> noting any qualifying files that don't have a valid /*TEST .. TEST*/
> block.

   This will be added, thanks for the suggestion.

> 
> On Fri, Jan 13, 2017 at 11:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>   PETSc developers,
>> 
>>     Scott Kruger has been implementing a new "test harness" for PETSc. Rather than having tests implemented as bash lines in the makefiles, the tests will be defined within the source code for each example with a simple format and the actual scripts that run the tests will be automatically generated. This offers three advantages over the current system
>> 
>> 1) test definitions can be given in a very concise format that is independent of how the test is run
>> 2) multiple approaches to actually running the tests can be implemented without requiring any changes to the test definitions for each example.
>> 3) the tests can be compiled and run in parallel (eventually we hope to see much faster times for testing).
>> 
>>   Here is an example of the format that defines tests
>> 
>>     test:
>> 
>>   test:
>>      suffix: yaml
>>      requires: yaml !complex
>>      args: -options_file_yaml bag.yml -options_view
>>      filter:  grep -v saws_port_auto_select
>>      localrunfiles: bag.yml
>> 
>> The first is a test that takes no command line arguments and does no filtering of output. The second takes command line arguments, does some filtering of the output and this example also requires a data file associated with that example.  The expected output of tests continues to reside in the output directory with the previously naming convention of XXX_1.out XXX_2.out where XXX is the name of the example or XXX_suffix.out where suffix is defined in the test definition.
>> 
>>  So far in the master branch we have only converted over two directories of examples sys/examples/tutorials and sys/examples/tests so you can look in those directories for other examples. Over the next few weeks we hope to convert over all the rest of the directories.
>> 
>>  Special thanks to Scott, Satish, and Jed for all the work they have done to set up the new test harness.
>> 
>>   Barry
>> 
>>   Technical details: the new python script ./config/gmakegentest.py which is normally automatically run at configure walks through the examples directories looking for the formatted test definitions, for each test it generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or tutorials/ a shell script that can run that test. In addition config/gmakegentest.py  adds dependencies to the $PETSC_DIR/gmakefile for all the tests that it has found. The rule make alltests then executes all those dependencies.
>> 
>>   We will be refining both the testing harness and the test definitions as we convert the rest of the examples over to the new approach.
>> 
>> 
>> 
>> 
>>