[petsc-dev] Introducing new "test harness" to PETSc

[Barry Smith]

  I still don't understand. Can you point to a test and a subtest among the directories that have been converted so far. Or have no subtests been converted yet?


> On Jan 24, 2017, at 2:56 PM, Scott Kruger <kruger at txcorp.com> wrote:
> 
> 
> 
> On 1/24/17 9:57 AM, Barry Smith wrote:
>> 
>>  I don't really understand what a "sub-test" is, just any test in an example that is not first one? Aside from the suffix are "sub-tests" handled any different than the first test? I don't think they should be.
> 
> subtest has a one-to-one correspondence to an mpiexec call
> A test has a one-to-one correspondence to a shell script and make target.
> 
> For-loops that invoke mpiexec call multiple times => multiple sub-tests.
> There are also just multiple invocations by hand.
> 
> How to handle diffs?
> 
> Ultimately, it was decided to have one output file per shell script
> to avoid making things too confusing.
> 
> convertExamples.py handles most of these "reasonably" well.
> 
> Scott
> 
> 
>> 
>>> On Jan 24, 2017, at 10:10 AM, Scott Kruger <kruger at txcorp.com> wrote:
>>> 
>>> 	
>>> 
>>> On 1/23/17 11:23 PM, Tobin Isaac wrote:
>>>> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
>>>> in zsh and bash but got nothing.
>>> 
>>> Good question.   I think this is handled by the bash-completion project
>>> which is normally an apt-get install or brew install, but has never
>>> worked on my mac for some reason.
>>> 
>>>> 
>>>> Thanks to everyone who worked on this, I just have two other
>>>> questions/requests:
>>>> 
>>>> - What’s the best way to update the test definitions after fiddling
>>>> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
>>>> automatically when sources are newer than their tests.
>>> 
>>> I'll let Jed comment on this.  It's not trivial with the current method
>>> of specifying targets (gmake's lisp-like language), and even my
>>> first pedantic implementation (explicitly writing out targets for
>>> each test), I seem to recall it being brittle because of the global
>>> nature of the gmakegentest.py requiring the config.
>>> 
>>> My development workflow is generally:
>>> - edit source file
>>> - run gmakegentest.py
>>> - inspect shell script in $PETSC_ARCH/tests
>>> - run shell script by hand and edit as needed
>>> - test with makefile (either search or global) before commit
>>> 
>>> There is also config/testparse.py which can be run on an
>>> individual file if you just want to test the parsing:
>>> 
>>>> $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
>>> ex1.c
>>>  runex1
>>>     filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error Created)" | cut -f1,2,3,4,5,6 -d" "
>>> 
>>>> 
>>>> - Is there a technical reason suffixes can only be specified at the
>>>> top level?  It would help with concision if subtest suffixes could
>>>> append to their parent suffixes.
>>> 
>>> Design decision on what should be a test an what should be a subtest.
>>> If you want your test to go to a different file, then it is now a test.
>>> Trying to do something more complicated would add
>>> unnecessary complexity.  If you do want some grouping, then it can
>>> be done with suffix itself; i.e., suffix does not have to be an
>>> integer, but can be an arbitrary string.
>>> 
>>> But perhaps I don't understand your use case.
>>> 
>>> Scott
>>> 
>>> 
>>> 
>>>> 
>>>> Cheers,
>>>> Toby
>>>> 
>>>> *From:* Jed Brown <mailto:jed at jedbrown.org>
>>>> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
>>>> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
>>>> <mailto:bsmith at mcs.anl.gov>
>>>> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>>>> 
>>>> Matthew Knepley <knepley at gmail.com> writes:
>>>>> Gripes:
>>>>> 
>>>>> 1) How do I run an individual test run?
>>>> 
>>>> make -f gmakefile sys_tutorials-runex5
>>>> 
>>>> (tab completion works) or use a search that matches only that run.
>>>> 
>>>>> I use
>>>>> 
>>>>> make -f gmakefile test searchin=plex
>>>>> 
>>>>> to run plex tests, and I think
>>>>> 
>>>>> make -f gmakefile test searchin=plex*ex1
>>>> 
>>>> Due to make syntax, % is the internal string matcher.
>>>> 
>>>>> will work. However, how do I select ex1_2?
>>>> 
>>>> make -f gmakefile test search=plex%ex1_2
>>>> 
>>>>> 2) How do I replace the output which exists with the output of the
>>>>> current test run?
>>>>> 
>>>>> This is amazingly useful after changes.
>>>> 
>>>> I have this script ~/bin/diffupdate:
>>>> #!/bin/sh
>>>> 
>>>> args=
>>>> if [ "$1" = "-u" ]; then
>>>>   args="-u"
>>>>   shift
>>>> fi
>>>> new="$2"
>>>> ref="$1"
>>>> 
>>>> diff $args "$ref" "$new"
>>>> ret=$?
>>>> echo "Updating $new --> $ref"
>>>> mv "$new" "$ref"
>>>> exit $ret
>>>> 
>>>> 
>>>> Then I run
>>>> 
>>>> make -f gmakefile test search=plex% DIFF=diffupdate
>>>> 
>>>> It prints the diff and replaces the reference file (first argument) with
>>>> the second.
>>>> 
>>>>> 3) How do I give extra arguments with the make interface, rather than
>>>>> test_harness?
>>>> 
>>>> That is, run some modification of the test and still do the diff, but
>>>> don't change the reference output?  I have used PETSC_OPTIONS for that.
>>>> 
>>>> make .... PETSC_OPTIONS=-more_magic
>>>> 
>>>>> Once I have all the functionality of my Python stuff,  I will throw it
>>>> away.
>>>> 
>>> 
>>> --
>>> Tech-X Corporation               kruger at txcorp.com
>>> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
>>> Boulder, CO 80303                Fax:   (303) 448-7756
>> 
> 
> -- 
> Tech-X Corporation               kruger at txcorp.com
> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
> Boulder, CO 80303                Fax:   (303) 448-7756