[petsc-dev] test harness with alternative output files

[Scott Kruger]

On 10/25/17 10:04 AM, Smith, Barry F. wrote:
> 
>     Scott,
> 
>      It looks like you started to put in support for the test harness to compare output against multiple files.
> 
> $ git grep altfile
> config/gmakegentest.py:    if len(altlist)>1: subst['altfiles']=altlist
> config/gmakegentest.py:      if 'altfiles' not in subst:
> config/gmakegentest.py:        for i in range(len(subst['altfiles'])):
> config/gmakegentest.py:          af=subst['altfiles'][i]
> config/gmakegentest.py:          if i!=len(subst['altfiles'])-1:
> 
> but that there is not yet support for this?

There is support but:
1. It requires the multiple files to be of the form ex*_[1-3]_alt.out
    in the output directory
2. It is piping all of the diffs simultaneously to create a single
    diff test rather than multple.

You can look at a $PETSC_ARCH/tests/src/sys/examples/tests/runex2.sh to 
see how this works.

I assuming that you would rather:
1. Have an altfiles (or similar) keyword (*)
2. Have separate difftests for each.

Scott

(*) see documentation or the top of config/testparse.py for accepted 
keywords currently
> 
> We have a bunch of tests that require this so we will need it in the test harness. How difficult do you think this would be to add?
> 
>     Thanks
> 
>     Barry
> 

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756