[petsc-dev] test harness with alternative output files

[Smith, Barry F.]

  Scott,

    Thanks. Might be enough as is.

   Satish,

     Could you see if this support is enough to fix he pesky example in master nightly that produces two different possible output.

   Barry

> On Oct 26, 2017, at 10:48 PM, Scott Kruger <kruger at txcorp.com> wrote:
> 
> 
> 
> On 10/25/17 10:04 AM, Smith, Barry F. wrote:
>>    Scott,
>>     It looks like you started to put in support for the test harness to compare output against multiple files.
>> $ git grep altfile
>> config/gmakegentest.py:    if len(altlist)>1: subst['altfiles']=altlist
>> config/gmakegentest.py:      if 'altfiles' not in subst:
>> config/gmakegentest.py:        for i in range(len(subst['altfiles'])):
>> config/gmakegentest.py:          af=subst['altfiles'][i]
>> config/gmakegentest.py:          if i!=len(subst['altfiles'])-1:
>> but that there is not yet support for this?
> 
> There is support but:
> 1. It requires the multiple files to be of the form ex*_[1-3]_alt.out
>   in the output directory
> 2. It is piping all of the diffs simultaneously to create a single
>   diff test rather than multple.
> 
> You can look at a $PETSC_ARCH/tests/src/sys/examples/tests/runex2.sh to see how this works.
> 
> I assuming that you would rather:
> 1. Have an altfiles (or similar) keyword (*)
> 2. Have separate difftests for each.
> 
> Scott
> 
> (*) see documentation or the top of config/testparse.py for accepted keywords currently
>> We have a bunch of tests that require this so we will need it in the test harness. How difficult do you think this would be to add?
>>    Thanks
>>    Barry
> 
> -- 
> Tech-X Corporation               kruger at txcorp.com
> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
> Boulder, CO 80303                Fax:   (303) 448-7756