[petsc-dev] Introducing new "test harness" to PETSc

[Matthew Knepley]

On Tue, Jan 17, 2017 at 4:19 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Jan 13, 2017 at 4:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>    PETSc developers,
>>
>>      Scott Kruger has been implementing a new "test harness" for PETSc.
>> Rather than having tests implemented as bash lines in the makefiles, the
>> tests will be defined within the source code for each example with a simple
>> format and the actual scripts that run the tests will be automatically
>> generated. This offers three advantages over the current system
>>
>> 1) test definitions can be given in a very concise format that is
>> independent of how the test is run
>> 2) multiple approaches to actually running the tests can be implemented
>> without requiring any changes to the test definitions for each example.
>> 3) the tests can be compiled and run in parallel (eventually we hope to
>> see much faster times for testing).
>>
>>    Here is an example of the format that defines tests
>>
>>      test:
>>
>>    test:
>>       suffix: yaml
>>       requires: yaml !complex
>>       args: -options_file_yaml bag.yml -options_view
>>       filter:  grep -v saws_port_auto_select
>>       localrunfiles: bag.yml
>>
>
> How do I specify the number of processes to use for the test? Or where
> would I look this up?
>

Also, how do I put comments inside these blocks to tell me what the test is
about?

   Matt


>   Thanks,
>
>      Matt
>
>
>> The first is a test that takes no command line arguments and does no
>> filtering of output. The second takes command line arguments, does some
>> filtering of the output and this example also requires a data file
>> associated with that example.  The expected output of tests continues to
>> reside in the output directory with the previously naming convention of
>> XXX_1.out XXX_2.out where XXX is the name of the example or XXX_suffix.out
>> where suffix is defined in the test definition.
>>
>>   So far in the master branch we have only converted over two directories
>> of examples sys/examples/tutorials and sys/examples/tests so you can look
>> in those directories for other examples. Over the next few weeks we hope to
>> convert over all the rest of the directories.
>>
>>   Special thanks to Scott, Satish, and Jed for all the work they have
>> done to set up the new test harness.
>>
>>    Barry
>>
>>    Technical details: the new python script ./config/gmakegentest.py
>> which is normally automatically run at configure walks through the examples
>> directories looking for the formatted test definitions, for each test it
>> generates in the directory $PETSC_ARCH/test/src/XXX/examples/tests or
>> tutorials/ a shell script that can run that test. In addition
>> config/gmakegentest.py  adds dependencies to the $PETSC_DIR/gmakefile for
>> all the tests that it has found. The rule make alltests then executes all
>> those dependencies.
>>
>>    We will be refining both the testing harness and the test definitions
>> as we convert the rest of the examples over to the new approach.
>>
>>
>>
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170117/30b90770/attachment.html>