[petsc-dev] Testing seems broken in master

[Satish Balay]

hm - indices are integers  and its not ignored (i.e %d) - only the %f and %e diff is ignored (by default)

Satish

On Mon, 4 Jan 2021, Scott Kruger wrote:

> 
> 
> 
> Is this just the 3rd problem?
> 
> Regarding how you can end up with changes not being caught:
> The default (going all the way back to the old harness) is to not check
> numbers to avoid round-off errors giving false negatives (failures).
> Of course, sometimes you *want* to check the numbers; e.g., for indices.  The
> solution for this is to add:
> 
>      diff_args: -j
> 
> Scott
> 
> On 12/31/20 4:40 PM, Barry Smith wrote:
> >
> >   I think I have it "fixed" now in the branch, once it passes the pipeline I
> > will shepard it through the MR quickly. Sorry about this, even all our CI
> > testing can miss a great deal.
> >
> >   Barry
> >
> >
> >
> >> On Dec 31, 2020, at 2:44 PM, Barry Smith <bsmith at petsc.dev
> >> <mailto:bsmith at petsc.dev>> wrote:
> >>
> >>
> >>   This is a different (3rd) problem. Funny it didn't bother anyone for two
> >> months.
> >>
> >>   Fix is in barry/2020-12-29/fix-petscdiff-bracket but the pipeline keeps
> >> failing ts_tutorials_advection-diffusion-reaction-ex3_2 fails on different
> >> machines with slightly different counts. I don't see how this change could
> >> cause that! But gets old results on my machine. Very frustrating.
> >> Barry
> >>
> >>> On Dec 31, 2020, at 1:02 PM, Matthew Knepley <knepley at gmail.com
> >>> <mailto:knepley at gmail.com>> wrote:
> >>>
> >>> On Thu, Dec 31, 2020 at 1:48 PM Barry Smith <bsmith at petsc.dev
> >>> <mailto:bsmith at petsc.dev>> wrote:
> >>>
> >>>
> >>>       So the programs output changes and should no longer match that
> >>>     in the output/* file yet the test harness does not error with a
> >>>     statement that the two outputs do not match?
> >>>
> >>>        I noticed the gmakegentest.py is not being run before it runs
> >>>     the test? Does this mean it is just running all the old stuff
> >>>     which does match fine?
> >>>
> >>>        Then either how petscdiff is called by the test harness has
> >>>     changed or petscdiff has changed and does not detect changes
> >>>     anymore
> >>>
> >>>        BTW: I always use -f ./gmakefile.test test not just the gmakefile
> >>>
> >>>        All the PETSc changes are trivial and can be seen with a
> >>>     simple diff, it is hard to believe they would cause this
> >>>     behavior but I guess they must.
> >>>
> >>>        You can go to PETSC_ARCH/tests/snes/tests and run the ex13
> >>>     shell script directly.
> >>>
> >>>
> >>> It is the sed problem:
> >>>
> >>> master *$:/PETSc3/petsc/petsc-dev$
> >>> /PETSc3/petsc/petsc-dev/lib/petsc/bin/petscdiff
> >>> /PETSc3/petsc/petsc-dev/src/snes/tests/output/ex13_bench.out
> >>> ex13_bench.tmp
> >>>
> >>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
> >>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
> >>> sed: 1: "s/\033[1;31m//g": unbalanced brackets ([])
> >>> sed: 1: "s/\033[0;39m\033[0;49m//g": unbalanced brackets ([])
> >>>
> >>> The error was getting eaten.
> >>>
> >>> This is in current master. Is it fixed in a branch?
> >>>
> >>>    Matt
> >>>
> >>>       Barry
> >>>
> >>>
> >>>>     On Dec 31, 2020, at 12:38 PM, Matthew Knepley
> >>>>     <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
> >>>>
> >>>>     I just pulled master, and simple alterations to tests do not
> >>>>     produce a failure:
> >>>>
> >>>>     master *$:/PETSc3/petsc/petsc-dev$ PETSC_ARCH=arch-master-debug
> >>>>     make -f ./gmakefile test search="snes_tests-ex13_bench"
> >>>>     TIMEOUT=5000 EXTRA_OPTIONS="-dm_
> >>>>     refine 0"
> >>>>     Using MAKEFLAGS: EXTRA_OPTIONS=-dm_refine 0 TIMEOUT=5000
> >>>>     search=snes_tests-ex13_bench
> >>>>             TEST
> >>>>     arch-master-debug/tests/counts/snes_tests-ex13_bench.counts
> >>>>      ok snes_tests-ex13_bench
> >>>>      ok diff-snes_tests-ex13_bench
> >>>>
> >>>>     I check that the runs produce different output when done manually.
> >>>>
> >>>>     Scott and Barry, could this be related to changed to testing?
> >>>>
> >>>>       Thanks,
> >>>>
> >>>>          Matt
> >>>>
> >>>>     -- 
> >>>>     What most experimenters take for granted before they begin
> >>>>     their experiments is infinitely more interesting than any
> >>>>     results to which their experiments lead.
> >>>>     -- Norbert Wiener
> >>>>
> >>>>     https://www.cse.buffalo.edu/~knepley/
> >>>>     <http://www.cse.buffalo.edu/%7Eknepley/>
> >>>
> >>>
> >>>
> >>> -- 
> >>> What most experimenters take for granted before they begin their
> >>> experiments is infinitely more interesting than any results to which their
> >>> experiments lead.
> >>> -- Norbert Wiener
> >>>
> >>> https://www.cse.buffalo.edu/~knepley/
> >>> <http://www.cse.buffalo.edu/%7Eknepley/>
> >>
> >
> 
>