[petsc-dev] Introducing new "test harness" to PETSc

[Barry Smith]

> On Jan 17, 2017, at 8:54 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Matthew Knepley <knepley at gmail.com> writes:
>> Gripes:
>> 
>>  1) How do I run an individual test run?
> 
>  make -f gmakefile sys_tutorials-runex5
> 
> (tab completion works) or use a search that matches only that run.
> 
>> I use
>> 
>>  make -f gmakefile test searchin=plex
>> 
>> to run plex tests, and I think
>> 
>>  make -f gmakefile test searchin=plex*ex1
> 
> Due to make syntax, % is the internal string matcher.
> 
>> will work. However, how do I select ex1_2?
> 
>  make -f gmakefile test search=plex%ex1_2
> 
>>  2) How do I replace the output which exists with the output of the
>> current test run?
>> 
>> This is amazingly useful after changes.
> 
> I have this script ~/bin/diffupdate:
> #!/bin/sh
> 
> args=
> if [ "$1" = "-u" ]; then
>    args="-u"
>    shift
> fi
> new="$2"
> ref="$1"
> 
> diff $args "$ref" "$new"
> ret=$?
> echo "Updating $new --> $ref"
> mv "$new" "$ref"
> exit $ret
> 
> 
> Then I run
> 
>  make -f gmakefile test search=plex% DIFF=diffupdate
> 
> It prints the diff and replaces the reference file (first argument) with
> the second.

  Unfortunately I don't have online access to your ~/bin/diffupdate.

  This needs to be a feature of the test harness directly.
> 
>>  3) How do I give extra arguments with the make interface, rather than
>> test_harness?
> 
> That is, run some modification of the test and still do the diff, but
> don't change the reference output?  I have used PETSC_OPTIONS for that.
> 
>  make .... PETSC_OPTIONS=-more_magic
> 
>> Once I have all the functionality of my Python stuff,  I will throw it away.
>