[petsc-dev] Introducing new "test harness" to PETSc

[Barry Smith]

  I don't really understand what a "sub-test" is, just any test in an example that is not first one? Aside from the suffix are "sub-tests" handled any different than the first test? I don't think they should be.


> On Jan 24, 2017, at 10:10 AM, Scott Kruger <kruger at txcorp.com> wrote:
> 
> 	
> 
> On 1/23/17 11:23 PM, Tobin Isaac wrote:
>> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
>> in zsh and bash but got nothing.
> 
> Good question.   I think this is handled by the bash-completion project
> which is normally an apt-get install or brew install, but has never
> worked on my mac for some reason.
> 
>> 
>> Thanks to everyone who worked on this, I just have two other
>> questions/requests:
>> 
>>  - What’s the best way to update the test definitions after fiddling
>> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
>> automatically when sources are newer than their tests.
> 
> I'll let Jed comment on this.  It's not trivial with the current method
> of specifying targets (gmake's lisp-like language), and even my
> first pedantic implementation (explicitly writing out targets for
> each test), I seem to recall it being brittle because of the global
> nature of the gmakegentest.py requiring the config.
> 
> My development workflow is generally:
> - edit source file
> - run gmakegentest.py
> - inspect shell script in $PETSC_ARCH/tests
> - run shell script by hand and edit as needed
> - test with makefile (either search or global) before commit
> 
> There is also config/testparse.py which can be run on an
> individual file if you just want to test the parsing:
> 
> > $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
> ex1.c
>   runex1
>      filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error Created)" | cut -f1,2,3,4,5,6 -d" "
> 
>> 
>>  - Is there a technical reason suffixes can only be specified at the
>> top level?  It would help with concision if subtest suffixes could
>> append to their parent suffixes.
> 
> Design decision on what should be a test an what should be a subtest.
> If you want your test to go to a different file, then it is now a test.
> Trying to do something more complicated would add
> unnecessary complexity.  If you do want some grouping, then it can
> be done with suffix itself; i.e., suffix does not have to be an
> integer, but can be an arbitrary string.
> 
> But perhaps I don't understand your use case.
> 
> Scott
> 
> 
> 
>> 
>> Cheers,
>>  Toby
>> 
>> *From:* Jed Brown <mailto:jed at jedbrown.org>
>> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
>> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
>> <mailto:bsmith at mcs.anl.gov>
>> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>> 
>> Matthew Knepley <knepley at gmail.com> writes:
>>> Gripes:
>>> 
>>>  1) How do I run an individual test run?
>> 
>>  make -f gmakefile sys_tutorials-runex5
>> 
>> (tab completion works) or use a search that matches only that run.
>> 
>>> I use
>>> 
>>>  make -f gmakefile test searchin=plex
>>> 
>>> to run plex tests, and I think
>>> 
>>>  make -f gmakefile test searchin=plex*ex1
>> 
>> Due to make syntax, % is the internal string matcher.
>> 
>>> will work. However, how do I select ex1_2?
>> 
>>  make -f gmakefile test search=plex%ex1_2
>> 
>>>  2) How do I replace the output which exists with the output of the
>>> current test run?
>>> 
>>> This is amazingly useful after changes.
>> 
>> I have this script ~/bin/diffupdate:
>> #!/bin/sh
>> 
>> args=
>> if [ "$1" = "-u" ]; then
>>    args="-u"
>>    shift
>> fi
>> new="$2"
>> ref="$1"
>> 
>> diff $args "$ref" "$new"
>> ret=$?
>> echo "Updating $new --> $ref"
>> mv "$new" "$ref"
>> exit $ret
>> 
>> 
>> Then I run
>> 
>>  make -f gmakefile test search=plex% DIFF=diffupdate
>> 
>> It prints the diff and replaces the reference file (first argument) with
>> the second.
>> 
>>>  3) How do I give extra arguments with the make interface, rather than
>>> test_harness?
>> 
>> That is, run some modification of the test and still do the diff, but
>> don't change the reference output?  I have used PETSC_OPTIONS for that.
>> 
>>  make .... PETSC_OPTIONS=-more_magic
>> 
>>> Once I have all the functionality of my Python stuff,  I will throw it
>> away.
>> 
> 
> -- 
> Tech-X Corporation               kruger at txcorp.com
> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
> Boulder, CO 80303                Fax:   (303) 448-7756