[petsc-dev] Testing seems broken in master

[Scott Kruger]

Yes.  There are test cases where a cast is done so for those.

Scott


On 1/4/21 10:29 AM, Satish Balay wrote:
> hm - indices are integers  and its not ignored (i.e %d) - only the %f and %e diff is ignored (by default)
>
> Satish
>
> On Mon, 4 Jan 2021, Scott Kruger wrote:
>
>>
>>
>> Is this just the 3rd problem?
>>
>> Regarding how you can end up with changes not being caught:
>> The default (going all the way back to the old harness) is to not check
>> numbers to avoid round-off errors giving false negatives (failures).
>> Of course, sometimes you *want* to check the numbers; e.g., for indices.  The
>> solution for this is to add:
>>
>>       diff_args: -j
>>
>> Scott
>>
>> On 12/31/20 4:40 PM, Barry Smith wrote:
>>>    I think I have it "fixed" now in the branch, once it passes the pipeline I
>>> will shepard it through the MR quickly. Sorry about this, even all our CI
>>> testing can miss a great deal.
>>>
>>>    Barry
>>>
>>>
>>>
>>>> On Dec 31, 2020, at 2:44 PM, Barry Smith <bsmith at petsc.dev
>>>> <mailto:bsmith at petsc.dev>> wrote:
>>>>
>>>>
>>>>    This is a different (3rd) problem. Funny it didn't bother anyone for two
>>>> months.
>>>>
>>>>    Fix is in barry/2020-12-29/fix-petscdiff-bracket but the pipeline keeps
>>>> failing ts_tutorials_advection-diffusion-reaction-ex3_2 fails on different
>>>> machines with slightly different counts. I don't see how this change could
>>>> cause that! But gets old results on my machine. Very frustrating.
>>>> Barry
>>>>
>>>>> On Dec 31, 2020, at 1:02 PM, Matthew Knepley <knepley at gmail.com
>>>>> <mailto:knepley at gmail.com>> wrote:
>>>>>
>>>>> On Thu, Dec 31, 2020 at 1:48 PM Barry Smith <bsmith at petsc.dev
>>>>> <mailto:bsmith at petsc.dev>> wrote:
>>>>>
>>>>>
>>>>>        So the programs output changes and should no longer match that
>>>>>      in the output/* file yet the test harness does not error with a
>>>>>      statement that the two outputs do not match?
>>>>>
>>>>>         I noticed the gmakegentest.py is not being run before it runs
>>>>>      the test? Does this mean it is just running all the old stuff
>>>>>      which does match fine?
>>>>>
>>>>>         Then either how petscdiff is called by the test harness has
>>>>>      changed or petscdiff has changed and does not detect changes
>>>>>      anymore
>>>>>
>>>>>         BTW: I always use -f ./gmakefile.test test not just the gmakefile
>>>>>
>>>>>         All the PETSc changes are trivial and can be seen with a
>>>>>      simple diff, it is hard to believe they would cause this
>>>>>      behavior but I guess they must.
>>>>>
>>>>>         You can go to PETSC_ARCH/tests/snes/tests and run the ex13
>>>>>      shell script directly.
>>>>>
>>>>>
>>>>> It is the sed problem:
>>>>>
>>>>> master *$:/PETSc3/petsc/petsc-dev$
>>>>> /PETSc3/petsc/petsc-dev/lib/petsc/bin/petscdiff
>>>>> /PETSc3/petsc/petsc-dev/src/snes/tests/output/ex13_bench.out
>>>>> ex13_bench.tmp
>>>>>
>>>>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
>>>>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
>>>>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
>>>>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
>>>>>
>>>>> The error was getting eaten.
>>>>>
>>>>> This is in current master. Is it fixed in a branch?
>>>>>
>>>>>     Matt
>>>>>
>>>>>        Barry
>>>>>
>>>>>
>>>>>>      On Dec 31, 2020, at 12:38 PM, Matthew Knepley
>>>>>>      <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>>>>>>
>>>>>>      I just pulled master, and simple alterations to tests do not
>>>>>>      produce a failure:
>>>>>>
>>>>>>      master *$:/PETSc3/petsc/petsc-dev$ PETSC_ARCH=arch-master-debug
>>>>>>      make -f ./gmakefile test search="snes_tests-ex13_bench"
>>>>>>      TIMEOUT=5000 EXTRA_OPTIONS="-dm_
>>>>>>      refine 0"
>>>>>>      Using MAKEFLAGS: EXTRA_OPTIONS=-dm_refine 0 TIMEOUT=5000
>>>>>>      search=snes_tests-ex13_bench
>>>>>>              TEST
>>>>>>      arch-master-debug/tests/counts/snes_tests-ex13_bench.counts
>>>>>>       ok snes_tests-ex13_bench
>>>>>>       ok diff-snes_tests-ex13_bench
>>>>>>
>>>>>>      I check that the runs produce different output when done manually.
>>>>>>
>>>>>>      Scott and Barry, could this be related to changed to testing?
>>>>>>
>>>>>>        Thanks,
>>>>>>
>>>>>>           Matt
>>>>>>
>>>>>>      --
>>>>>>      What most experimenters take for granted before they begin
>>>>>>      their experiments is infinitely more interesting than any
>>>>>>      results to which their experiments lead.
>>>>>>      -- Norbert Wiener
>>>>>>
>>>>>>      https://www.cse.buffalo.edu/~knepley/
>>>>>>      <http://www.cse.buffalo.edu/%7Eknepley/>
>>>>>
>>>>>
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 466-3196
Boulder, CO 80303                Fax:   (303) 448-7756

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210104/913a227a/attachment.html>