[petsc-dev] Introducing new "test harness" to PETSc

[Scott Kruger]

>
> I don't know how easy it would be to implement, but having a syntax for substituting dictionary keys into value string would be really good, e.g.,
>
> ~~~~
> test:
>   pctype: {{sor,icc}}
>   output_file: output/ex1_ at pctype@.out
>   args: -ksp_monitor -pc_type @pctype@


Sorry -- forgot to comment on this.
I'm doing some more tests and will comment in a new thread.

Scott



> ~~~~
>
> Cheers,
>   Toby
>
> ________________________________________
> From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on behalf of Scott Kruger [kruger at txcorp.com]
> Sent: Tuesday, January 24, 2017 10:10
> Cc: petsc-dev
> Subject: Re: [petsc-dev] Introducing new "test harness" to PETSc
>
> On 1/23/17 11:23 PM, Tobin Isaac wrote:
>> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
>> in zsh and bash but got nothing.
>
> Good question.   I think this is handled by the bash-completion project
> which is normally an apt-get install or brew install, but has never
> worked on my mac for some reason.
>
>>
>> Thanks to everyone who worked on this, I just have two other
>> questions/requests:
>>
>>   - What’s the best way to update the test definitions after fiddling
>> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
>> automatically when sources are newer than their tests.
>
> I'll let Jed comment on this.  It's not trivial with the current method
> of specifying targets (gmake's lisp-like language), and even my
> first pedantic implementation (explicitly writing out targets for
> each test), I seem to recall it being brittle because of the global
> nature of the gmakegentest.py requiring the config.
>
> My development workflow is generally:
>   - edit source file
>   - run gmakegentest.py
>   - inspect shell script in $PETSC_ARCH/tests
>   - run shell script by hand and edit as needed
>   - test with makefile (either search or global) before commit
>
> There is also config/testparse.py which can be run on an
> individual file if you just want to test the parsing:
>
>  > $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
> ex1.c
>     runex1
>        filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error
> Created)" | cut -f1,2,3,4,5,6 -d" "
>
>>
>>   - Is there a technical reason suffixes can only be specified at the
>> top level?  It would help with concision if subtest suffixes could
>> append to their parent suffixes.
>
> Design decision on what should be a test an what should be a subtest.
> If you want your test to go to a different file, then it is now a test.
> Trying to do something more complicated would add
> unnecessary complexity.  If you do want some grouping, then it can
> be done with suffix itself; i.e., suffix does not have to be an
> integer, but can be an arbitrary string.
>
> But perhaps I don't understand your use case.
>
> Scott
>
>
>
>>
>> Cheers,
>>   Toby
>>
>> *From:* Jed Brown <mailto:jed at jedbrown.org>
>> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
>> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
>> <mailto:bsmith at mcs.anl.gov>
>> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
>>
>> Matthew Knepley <knepley at gmail.com> writes:
>>> Gripes:
>>>
>>>   1) How do I run an individual test run?
>>
>>   make -f gmakefile sys_tutorials-runex5
>>
>> (tab completion works) or use a search that matches only that run.
>>
>>> I use
>>>
>>>   make -f gmakefile test searchin=plex
>>>
>>> to run plex tests, and I think
>>>
>>>   make -f gmakefile test searchin=plex*ex1
>>
>> Due to make syntax, % is the internal string matcher.
>>
>>> will work. However, how do I select ex1_2?
>>
>>   make -f gmakefile test search=plex%ex1_2
>>
>>>   2) How do I replace the output which exists with the output of the
>>> current test run?
>>>
>>> This is amazingly useful after changes.
>>
>> I have this script ~/bin/diffupdate:
>> #!/bin/sh
>>
>> args=
>> if [ "$1" = "-u" ]; then
>>     args="-u"
>>     shift
>> fi
>> new="$2"
>> ref="$1"
>>
>> diff $args "$ref" "$new"
>> ret=$?
>> echo "Updating $new --> $ref"
>> mv "$new" "$ref"
>> exit $ret
>>
>>
>> Then I run
>>
>>   make -f gmakefile test search=plex% DIFF=diffupdate
>>
>> It prints the diff and replaces the reference file (first argument) with
>> the second.
>>
>>>   3) How do I give extra arguments with the make interface, rather than
>>> test_harness?
>>
>> That is, run some modification of the test and still do the diff, but
>> don't change the reference output?  I have used PETSC_OPTIONS for that.
>>
>>   make .... PETSC_OPTIONS=-more_magic
>>
>>> Once I have all the functionality of my Python stuff,  I will throw it
>> away.
>>
>
> --
> Tech-X Corporation               kruger at txcorp.com
> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
> Boulder, CO 80303                Fax:   (303) 448-7756
>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756