[petsc-dev] Introducing new "test harness" to PETSc

[Scott Kruger]

On 1/23/17 11:23 PM, Tobin Isaac wrote:
> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
> in zsh and bash but got nothing.

Good question.   I think this is handled by the bash-completion project
which is normally an apt-get install or brew install, but has never
worked on my mac for some reason.

>
> Thanks to everyone who worked on this, I just have two other
> questions/requests:
>
>   - What’s the best way to update the test definitions after fiddling
> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
> automatically when sources are newer than their tests.

I'll let Jed comment on this.  It's not trivial with the current method
of specifying targets (gmake's lisp-like language), and even my
first pedantic implementation (explicitly writing out targets for
each test), I seem to recall it being brittle because of the global
nature of the gmakegentest.py requiring the config.

My development workflow is generally:
  - edit source file
  - run gmakegentest.py
  - inspect shell script in $PETSC_ARCH/tests
  - run shell script by hand and edit as needed
  - test with makefile (either search or global) before commit

There is also config/testparse.py which can be run on an
individual file if you just want to test the parsing:

 > $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
ex1.c
    runex1
       filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error 
Created)" | cut -f1,2,3,4,5,6 -d" "

>
>   - Is there a technical reason suffixes can only be specified at the
> top level?  It would help with concision if subtest suffixes could
> append to their parent suffixes.

Design decision on what should be a test an what should be a subtest.
If you want your test to go to a different file, then it is now a test.
Trying to do something more complicated would add
unnecessary complexity.  If you do want some grouping, then it can
be done with suffix itself; i.e., suffix does not have to be an
integer, but can be an arbitrary string.

But perhaps I don't understand your use case.

Scott



>
> Cheers,
>   Toby
>
> *From:* Jed Brown <mailto:jed at jedbrown.org>
> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
> <mailto:bsmith at mcs.anl.gov>
> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>
> Matthew Knepley <knepley at gmail.com> writes:
>> Gripes:
>>
>>   1) How do I run an individual test run?
>
>   make -f gmakefile sys_tutorials-runex5
>
> (tab completion works) or use a search that matches only that run.
>
>> I use
>>
>>   make -f gmakefile test searchin=plex
>>
>> to run plex tests, and I think
>>
>>   make -f gmakefile test searchin=plex*ex1
>
> Due to make syntax, % is the internal string matcher.
>
>> will work. However, how do I select ex1_2?
>
>   make -f gmakefile test search=plex%ex1_2
>
>>   2) How do I replace the output which exists with the output of the
>> current test run?
>>
>> This is amazingly useful after changes.
>
> I have this script ~/bin/diffupdate:
> #!/bin/sh
>
> args=
> if [ "$1" = "-u" ]; then
>     args="-u"
>     shift
> fi
> new="$2"
> ref="$1"
>
> diff $args "$ref" "$new"
> ret=$?
> echo "Updating $new --> $ref"
> mv "$new" "$ref"
> exit $ret
>
>
> Then I run
>
>   make -f gmakefile test search=plex% DIFF=diffupdate
>
> It prints the diff and replaces the reference file (first argument) with
> the second.
>
>>   3) How do I give extra arguments with the make interface, rather than
>> test_harness?
>
> That is, run some modification of the test and still do the diff, but
> don't change the reference output?  I have used PETSC_OPTIONS for that.
>
>   make .... PETSC_OPTIONS=-more_magic
>
>> Once I have all the functionality of my Python stuff,  I will throw it
> away.
>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756