[petsc-dev] Introducing new "test harness" to PETSc

Tobin Isaac tisaac at uchicago.edu
Tue Jan 24 11:21:34 CST 2017

Thanks, Scott. (Sorry I'm not replying inline, I'm using a buggy webmail client ATM).

- Tab completion works for other targets in gmakefile, but not for the tests

- I stumbled into the same workflow, and it works fine

- The reason I'm interested in suffixes is that I like the idea of for-loops, but it doesn't look like there's a way to use the loop variable: I can't write `args: -pc_type {{sor,icc}}` and have them diff'ed against separate output files.  So barring that, it would be nice to have appending suffixes, so that I could have, e.g.,

  args: -ksp_monitor
    suffix: sor
    args: -pc_type sor
    suffix: icc
    args: -pc_type icc

I don't know how easy it would be to implement, but having a syntax for substituting dictionary keys into value string would be really good, e.g.,

  pctype: {{sor,icc}}
  output_file: output/ex1_ at pctype@.out
  args: -ksp_monitor -pc_type @pctype@

From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] on behalf of Scott Kruger [kruger at txcorp.com]
Sent: Tuesday, January 24, 2017 10:10
Cc: petsc-dev
Subject: Re: [petsc-dev] Introducing new "test harness" to PETSc

On 1/23/17 11:23 PM, Tobin Isaac wrote:
> How does tab completion work?  I've tried `make -f gmakefile sys<Tab>`
> in zsh and bash but got nothing.

Good question.   I think this is handled by the bash-completion project
which is normally an apt-get install or brew install, but has never
worked on my mac for some reason.

> Thanks to everyone who worked on this, I just have two other
> questions/requests:
>   - What’s the best way to update the test definitions after fiddling
> with /*TEST*/ blocks?  It would be nice if gmakegentests.py would run
> automatically when sources are newer than their tests.

I'll let Jed comment on this.  It's not trivial with the current method
of specifying targets (gmake's lisp-like language), and even my
first pedantic implementation (explicitly writing out targets for
each test), I seem to recall it being brittle because of the global
nature of the gmakegentest.py requiring the config.

My development workflow is generally:
  - edit source file
  - run gmakegentest.py
  - inspect shell script in $PETSC_ARCH/tests
  - run shell script by hand and edit as needed
  - test with makefile (either search or global) before commit

There is also config/testparse.py which can be run on an
individual file if you just want to test the parsing:

 > $PETSC_DIR/config/testparse.py -v 1 -t ex1.c
       filter: egrep "(PETSC ERROR)" | egrep "(main|CreateError|Error
Created)" | cut -f1,2,3,4,5,6 -d" "

>   - Is there a technical reason suffixes can only be specified at the
> top level?  It would help with concision if subtest suffixes could
> append to their parent suffixes.

Design decision on what should be a test an what should be a subtest.
If you want your test to go to a different file, then it is now a test.
Trying to do something more complicated would add
unnecessary complexity.  If you do want some grouping, then it can
be done with suffix itself; i.e., suffix does not have to be an
integer, but can be an arbitrary string.

But perhaps I don't understand your use case.


> Cheers,
>   Toby
> *From:* Jed Brown <mailto:jed at jedbrown.org>
> *Sent:* ‎Tuesday‎, ‎17‎ ‎January‎, ‎2017 ‎20‎:‎55
> *To:* Matthew Knepley <mailto:knepley at gmail.com>, Barry Smith
> <mailto:bsmith at mcs.anl.gov>
> *Cc:* petsc-dev <mailto:petsc-dev at mcs.anl.gov>
> Matthew Knepley <knepley at gmail.com> writes:
>> Gripes:
>>   1) How do I run an individual test run?
>   make -f gmakefile sys_tutorials-runex5
> (tab completion works) or use a search that matches only that run.
>> I use
>>   make -f gmakefile test searchin=plex
>> to run plex tests, and I think
>>   make -f gmakefile test searchin=plex*ex1
> Due to make syntax, % is the internal string matcher.
>> will work. However, how do I select ex1_2?
>   make -f gmakefile test search=plex%ex1_2
>>   2) How do I replace the output which exists with the output of the
>> current test run?
>> This is amazingly useful after changes.
> I have this script ~/bin/diffupdate:
> #!/bin/sh
> args=
> if [ "$1" = "-u" ]; then
>     args="-u"
>     shift
> fi
> new="$2"
> ref="$1"
> diff $args "$ref" "$new"
> ret=$?
> echo "Updating $new --> $ref"
> mv "$new" "$ref"
> exit $ret
> Then I run
>   make -f gmakefile test search=plex% DIFF=diffupdate
> It prints the diff and replaces the reference file (first argument) with
> the second.
>>   3) How do I give extra arguments with the make interface, rather than
>> test_harness?
> That is, run some modification of the test and still do the diff, but
> don't change the reference output?  I have used PETSC_OPTIONS for that.
>   make .... PETSC_OPTIONS=-more_magic
>> Once I have all the functionality of my Python stuff,  I will throw it
> away.

Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756

More information about the petsc-dev mailing list