[petsc-dev] Testing seems broken in master

[Scott Kruger]

Is this just the 3rd problem?

Regarding how you can end up with changes not being caught:
The default (going all the way back to the old harness) is to not check 
numbers to avoid round-off errors giving false negatives (failures).
Of course, sometimes you *want* to check the numbers; e.g., for 
indices.  The solution for this is to add:

      diff_args: -j

Scott

On 12/31/20 4:40 PM, Barry Smith wrote:
>
>   I think I have it "fixed" now in the branch, once it passes the 
> pipeline I will shepard it through the MR quickly. Sorry about this, 
> even all our CI testing can miss a great deal.
>
>   Barry
>
>
>
>> On Dec 31, 2020, at 2:44 PM, Barry Smith <bsmith at petsc.dev 
>> <mailto:bsmith at petsc.dev>> wrote:
>>
>>
>>   This is a different (3rd) problem. Funny it didn't bother anyone 
>> for two months.
>>
>>   Fix is in barry/2020-12-29/fix-petscdiff-bracket but the pipeline 
>> keeps failing ts_tutorials_advection-diffusion-reaction-ex3_2 fails 
>> on different machines with slightly different counts. I don't see how 
>> this change could cause that! But gets old results on my machine. 
>> Very frustrating.
>> Barry
>>
>>> On Dec 31, 2020, at 1:02 PM, Matthew Knepley <knepley at gmail.com 
>>> <mailto:knepley at gmail.com>> wrote:
>>>
>>> On Thu, Dec 31, 2020 at 1:48 PM Barry Smith <bsmith at petsc.dev 
>>> <mailto:bsmith at petsc.dev>> wrote:
>>>
>>>
>>>       So the programs output changes and should no longer match that
>>>     in the output/* file yet the test harness does not error with a
>>>     statement that the two outputs do not match?
>>>
>>>        I noticed the gmakegentest.py is not being run before it runs
>>>     the test? Does this mean it is just running all the old stuff
>>>     which does match fine?
>>>
>>>        Then either how petscdiff is called by the test harness has
>>>     changed or petscdiff has changed and does not detect changes
>>>     anymore
>>>
>>>        BTW: I always use -f ./gmakefile.test test not just the gmakefile
>>>
>>>        All the PETSc changes are trivial and can be seen with a
>>>     simple diff, it is hard to believe they would cause this
>>>     behavior but I guess they must.
>>>
>>>        You can go to PETSC_ARCH/tests/snes/tests and run the ex13
>>>     shell script directly.
>>>
>>>
>>> It is the sed problem:
>>>
>>> master *$:/PETSc3/petsc/petsc-dev$ 
>>> /PETSc3/petsc/petsc-dev/lib/petsc/bin/petscdiff 
>>> /PETSc3/petsc/petsc-dev/src/snes/tests/output/ex13_bench.out 
>>> ex13_bench.tmp
>>>
>>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
>>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
>>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
>>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
>>>
>>> The error was getting eaten.
>>>
>>> This is in current master. Is it fixed in a branch?
>>>
>>>    Matt
>>>
>>>       Barry
>>>
>>>
>>>>     On Dec 31, 2020, at 12:38 PM, Matthew Knepley
>>>>     <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>
>>>>     I just pulled master, and simple alterations to tests do not
>>>>     produce a failure:
>>>>
>>>>     master *$:/PETSc3/petsc/petsc-dev$ PETSC_ARCH=arch-master-debug
>>>>     make -f ./gmakefile test search="snes_tests-ex13_bench"
>>>>     TIMEOUT=5000 EXTRA_OPTIONS="-dm_
>>>>     refine 0"
>>>>     Using MAKEFLAGS: EXTRA_OPTIONS=-dm_refine 0 TIMEOUT=5000
>>>>     search=snes_tests-ex13_bench
>>>>             TEST
>>>>     arch-master-debug/tests/counts/snes_tests-ex13_bench.counts
>>>>      ok snes_tests-ex13_bench
>>>>      ok diff-snes_tests-ex13_bench
>>>>
>>>>     I check that the runs produce different output when done manually.
>>>>
>>>>     Scott and Barry, could this be related to changed to testing?
>>>>
>>>>       Thanks,
>>>>
>>>>          Matt
>>>>
>>>>     -- 
>>>>     What most experimenters take for granted before they begin
>>>>     their experiments is infinitely more interesting than any
>>>>     results to which their experiments lead.
>>>>     -- Norbert Wiener
>>>>
>>>>     https://www.cse.buffalo.edu/~knepley/
>>>>     <http://www.cse.buffalo.edu/%7Eknepley/>
>>>
>>>
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their 
>>> experiments is infinitely more interesting than any results to which 
>>> their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/ 
>>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 466-3196
Boulder, CO 80303                Fax:   (303) 448-7756

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210104/0ca40f52/attachment-0001.html>