[petsc-dev] Introducing new "test harness" to PETSc

[Scott Kruger]

On 1/24/17 9:57 AM, Barry Smith wrote:
>
>   I don't really understand what a "sub-test" is, just any test in an example that is not first one? Aside from the suffix are "sub-tests" handled any different than the first test? I don't think they should be.

subtest has a one-to-one correspondence to an mpiexec call
A test has a one-to-one correspondence to a shell script and make target.

For-loops that invoke mpiexec call multiple times => multiple sub-tests.
There are also just multiple invocations by hand.

How to handle diffs?

Ultimately, it was decided to have one output file per shell script
to avoid making things too confusing.

convertExamples.py handles most of these "reasonably" well.

Scott


>
>> On Jan 24, 2017, at 10:10 AM, Scott Kruger <kruger at txcorp.com> wrote:
>>
>> 	
>>
>> On 1/23/17 11:23 PM, Tobin Isaac wrote:
>>> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
>>> in zsh and bash but got nothing.
>>
>> Good question.   I think this is handled by the bash-completion project
>> which is normally an apt-get install or brew install, but has never
>> worked on my mac for some reason.
>>
>>>
>>> Thanks to everyone who worked on this, I just have two other
>>> questions/requests:
>>>
>>>  - What’s the best way to update the test definitions after fiddling
>>> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
>>> automatically when sources are newer than their tests.
>>
>> I'll let Jed comment on this.  It's not trivial with the current method
>> of specifying targets (gmake's lisp-like language), and even my
>> first pedantic implementation (explicitly writing out targets for
>> each test), I seem to recall it being brittle because of the global
>> nature of the gmakegentest.py requiring the config.
>>
>> My development workflow is generally:
>> - edit source file
>> - run gmakegentest.py
>> - inspect shell script in $PETSC_ARCH/tests
>> - run shell script by hand and edit as needed
>> - test with makefile (either search or global) before commit
>>
>> There is also config/testparse.py which can be run on an
>> individual file if you just want to test the parsing:
>>
>>> $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
>> ex1.c
>>   runex1
>>      filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error Created)" | cut -f1,2,3,4,5,6 -d" "
>>
>>>
>>>  - Is there a technical reason suffixes can only be specified at the
>>> top level?  It would help with concision if subtest suffixes could
>>> append to their parent suffixes.
>>
>> Design decision on what should be a test an what should be a subtest.
>> If you want your test to go to a different file, then it is now a test.
>> Trying to do something more complicated would add
>> unnecessary complexity.  If you do want some grouping, then it can
>> be done with suffix itself; i.e., suffix does not have to be an
>> integer, but can be an arbitrary string.
>>
>> But perhaps I don't understand your use case.
>>
>> Scott
>>
>>
>>
>>>
>>> Cheers,
>>>  Toby
>>>
>>> *From:* Jed Brown <mailto:jed at jedbrown.org>
>>> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
>>> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
>>> <mailto:bsmith at mcs.anl.gov>
>>> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>>>
>>> Matthew Knepley <knepley at gmail.com> writes:
>>>> Gripes:
>>>>
>>>>  1) How do I run an individual test run?
>>>
>>>  make -f gmakefile sys_tutorials-runex5
>>>
>>> (tab completion works) or use a search that matches only that run.
>>>
>>>> I use
>>>>
>>>>  make -f gmakefile test searchin=plex
>>>>
>>>> to run plex tests, and I think
>>>>
>>>>  make -f gmakefile test searchin=plex*ex1
>>>
>>> Due to make syntax, % is the internal string matcher.
>>>
>>>> will work. However, how do I select ex1_2?
>>>
>>>  make -f gmakefile test search=plex%ex1_2
>>>
>>>>  2) How do I replace the output which exists with the output of the
>>>> current test run?
>>>>
>>>> This is amazingly useful after changes.
>>>
>>> I have this script ~/bin/diffupdate:
>>> #!/bin/sh
>>>
>>> args=
>>> if [ "$1" = "-u" ]; then
>>>    args="-u"
>>>    shift
>>> fi
>>> new="$2"
>>> ref="$1"
>>>
>>> diff $args "$ref" "$new"
>>> ret=$?
>>> echo "Updating $new --> $ref"
>>> mv "$new" "$ref"
>>> exit $ret
>>>
>>>
>>> Then I run
>>>
>>>  make -f gmakefile test search=plex% DIFF=diffupdate
>>>
>>> It prints the diff and replaces the reference file (first argument) with
>>> the second.
>>>
>>>>  3) How do I give extra arguments with the make interface, rather than
>>>> test_harness?
>>>
>>> That is, run some modification of the test and still do the diff, but
>>> don't change the reference output?  I have used PETSC_OPTIONS for that.
>>>
>>>  make .... PETSC_OPTIONS=-more_magic
>>>
>>>> Once I have all the functionality of my Python stuff,  I will throw it
>>> away.
>>>
>>
>> --
>> Tech-X Corporation               kruger at txcorp.com
>> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
>> Boulder, CO 80303                Fax:   (303) 448-7756
>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756