[petsc-dev] Introducing new "test harness" to PETSc

[Scott Kruger]

Most of them are in mat and ksp/ksp.

See ksp/ksp/examples/tests:runex32_testset1 for an extreme for loop example.

See mat/examples/tests:runex2 for multiple subtests within a single target.

There are others in those makefiles as well (and tutorials dirs).

Needless to say, adding subtest functionality was somewhat challenging,
and automating the conversion even more so.  I hope folks find it useful!

Scott


On 1/24/17 2:00 PM, Barry Smith wrote:
>
>   I still don't understand. Can you point to a test and a subtest among the directories that have been converted so far. Or have no subtests been converted yet?
>
>
>> On Jan 24, 2017, at 2:56 PM, Scott Kruger <kruger at txcorp.com> wrote:
>>
>>
>>
>> On 1/24/17 9:57 AM, Barry Smith wrote:
>>>
>>>  I don't really understand what a "sub-test" is, just any test in an example that is not first one? Aside from the suffix are "sub-tests" handled any different than the first test? I don't think they should be.
>>
>> subtest has a one-to-one correspondence to an mpiexec call
>> A test has a one-to-one correspondence to a shell script and make target.
>>
>> For-loops that invoke mpiexec call multiple times => multiple sub-tests.
>> There are also just multiple invocations by hand.
>>
>> How to handle diffs?
>>
>> Ultimately, it was decided to have one output file per shell script
>> to avoid making things too confusing.
>>
>> convertExamples.py handles most of these "reasonably" well.
>>
>> Scott
>>
>>
>>>
>>>> On Jan 24, 2017, at 10:10 AM, Scott Kruger <kruger at txcorp.com> wrote:
>>>>
>>>> 	
>>>>
>>>> On 1/23/17 11:23 PM, Tobin Isaac wrote:
>>>>> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
>>>>> in zsh and bash but got nothing.
>>>>
>>>> Good question.   I think this is handled by the bash-completion project
>>>> which is normally an apt-get install or brew install, but has never
>>>> worked on my mac for some reason.
>>>>
>>>>>
>>>>> Thanks to everyone who worked on this, I just have two other
>>>>> questions/requests:
>>>>>
>>>>> - What’s the best way to update the test definitions after fiddling
>>>>> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
>>>>> automatically when sources are newer than their tests.
>>>>
>>>> I'll let Jed comment on this.  It's not trivial with the current method
>>>> of specifying targets (gmake's lisp-like language), and even my
>>>> first pedantic implementation (explicitly writing out targets for
>>>> each test), I seem to recall it being brittle because of the global
>>>> nature of the gmakegentest.py requiring the config.
>>>>
>>>> My development workflow is generally:
>>>> - edit source file
>>>> - run gmakegentest.py
>>>> - inspect shell script in $PETSC_ARCH/tests
>>>> - run shell script by hand and edit as needed
>>>> - test with makefile (either search or global) before commit
>>>>
>>>> There is also config/testparse.py which can be run on an
>>>> individual file if you just want to test the parsing:
>>>>
>>>>> $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
>>>> ex1.c
>>>>  runex1
>>>>     filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error Created)" | cut -f1,2,3,4,5,6 -d" "
>>>>
>>>>>
>>>>> - Is there a technical reason suffixes can only be specified at the
>>>>> top level?  It would help with concision if subtest suffixes could
>>>>> append to their parent suffixes.
>>>>
>>>> Design decision on what should be a test an what should be a subtest.
>>>> If you want your test to go to a different file, then it is now a test.
>>>> Trying to do something more complicated would add
>>>> unnecessary complexity.  If you do want some grouping, then it can
>>>> be done with suffix itself; i.e., suffix does not have to be an
>>>> integer, but can be an arbitrary string.
>>>>
>>>> But perhaps I don't understand your use case.
>>>>
>>>> Scott
>>>>
>>>>
>>>>
>>>>>
>>>>> Cheers,
>>>>> Toby
>>>>>
>>>>> *From:* Jed Brown <mailto:jed at jedbrown.org>
>>>>> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
>>>>> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
>>>>> <mailto:bsmith at mcs.anl.gov>
>>>>> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>>>>>
>>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>>> Gripes:
>>>>>>
>>>>>> 1) How do I run an individual test run?
>>>>>
>>>>> make -f gmakefile sys_tutorials-runex5
>>>>>
>>>>> (tab completion works) or use a search that matches only that run.
>>>>>
>>>>>> I use
>>>>>>
>>>>>> make -f gmakefile test searchin=plex
>>>>>>
>>>>>> to run plex tests, and I think
>>>>>>
>>>>>> make -f gmakefile test searchin=plex*ex1
>>>>>
>>>>> Due to make syntax, % is the internal string matcher.
>>>>>
>>>>>> will work. However, how do I select ex1_2?
>>>>>
>>>>> make -f gmakefile test search=plex%ex1_2
>>>>>
>>>>>> 2) How do I replace the output which exists with the output of the
>>>>>> current test run?
>>>>>>
>>>>>> This is amazingly useful after changes.
>>>>>
>>>>> I have this script ~/bin/diffupdate:
>>>>> #!/bin/sh
>>>>>
>>>>> args=
>>>>> if [ "$1" = "-u" ]; then
>>>>>   args="-u"
>>>>>   shift
>>>>> fi
>>>>> new="$2"
>>>>> ref="$1"
>>>>>
>>>>> diff $args "$ref" "$new"
>>>>> ret=$?
>>>>> echo "Updating $new --> $ref"
>>>>> mv "$new" "$ref"
>>>>> exit $ret
>>>>>
>>>>>
>>>>> Then I run
>>>>>
>>>>> make -f gmakefile test search=plex% DIFF=diffupdate
>>>>>
>>>>> It prints the diff and replaces the reference file (first argument) with
>>>>> the second.
>>>>>
>>>>>> 3) How do I give extra arguments with the make interface, rather than
>>>>>> test_harness?
>>>>>
>>>>> That is, run some modification of the test and still do the diff, but
>>>>> don't change the reference output?  I have used PETSC_OPTIONS for that.
>>>>>
>>>>> make .... PETSC_OPTIONS=-more_magic
>>>>>
>>>>>> Once I have all the functionality of my Python stuff,  I will throw it
>>>>> away.
>>>>>
>>>>
>>>> --
>>>> Tech-X Corporation               kruger at txcorp.com
>>>> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
>>>> Boulder, CO 80303                Fax:   (303) 448-7756
>>>
>>
>> --
>> Tech-X Corporation               kruger at txcorp.com
>> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
>> Boulder, CO 80303                Fax:   (303) 448-7756
>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756