[petsc-dev] new test harness in PETSc

[Patrick Sanan]

Awesome!

As a contributor, what I am I supposed to do when a test fails due to some
error of mine?
What's the procedure to be able to run a test "under my own control" (be
able to see what's actually being executed, and what's failing in the diff)
so that I can fix my error?
That is, I want to be able to "back out of the test harness" to be able to
debug a failure myself.

This following is what I would try, given my limited knowledge of how
things work and after a quick scan of the docs. Is this as intended or is
there an easier way?

--------------------------------------------------

I make some seemingly-innocuous changes to the code. (In this case I
literally change nothing in master.)

I want to run all the tests to make sure I didn't break something that I
don't understand, so I look at the User Manual and copy and run this
command:

   make -f gmakefile.test test

I go away for a while and come back and see that some (3) tests have
failed, and that the harness gives me instructions on how to re-run them.

    # -------------
    #   Summary
    # -------------
    # FAILED ts_tutorials-ex11_adv_2d_quad_0
diff-sys_classes_viewer_tests-ex4_4 ts_tutorials-ex11_adv_2d_quad_1
    # success 3051/3915 tests (77.9%)
    # failed 3/3915 tests (0.1%)
    # todo 91/3915 tests (2.3%)
    # skip 770/3915 tests (19.7%)
    #
    # To rerun failed tests:
    #     /opt/local/bin/gmake -f gmakefile test
search='ts_tutorials-runex11_adv_2d_quad_0
sys_classes_viewer_tests-runex4_4 ts_tutorials-runex11_adv_2d_quad'

I want to take a look at the output so I can see what's happening - I see
that I'm given instructions on how to get more verbose output, so I run one
of the tests again with V=1 :

    /opt/local/bin/gmake V=1 -f gmakefile test
search='ts_tutorials-runex11_adv_2d_quad_0'

This tells me that this is failing because of a timeout

    # Exceeded timeout limit of 60 s

I poke around in the dev manual (note that "make -f gmakefile help" and
"make -f gmakefile.test help" say nothing about TIMEOUT as promised)
and eventually run

     /opt/local/bin/gmake V=1 TIMEOUT=1000 -f gmakefile test
search='ts_tutorials-runex11_adv_2d_quad_0'

and get success. Sweet - my assumption that this failed because I'm running
a debug build on a slow laptop is confirmed.

    # -------------
    #   Summary
    # -------------
   # success 2/2 tests (100.0%)
    # failed 0/2 tests (0.0%)
    # todo 0/2 tests (0.0%)
    # skip 0/2 tests (0.0%)

I then look at the other failed test.

     /opt/local/bin/gmake V=1 -f gmakefile test
search='sys_classes_viewer_tests-runex4_4'

This one seems to be failing because of the diff:


arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh
 -v
    ok sys_classes_viewer_tests-ex4_4 cat ex4_4.tmp | > /dev/null; cat
ex4a1.tmp > ex4_4.tmp.tmp 2>> runex4_4.err && mv ex4_4.tmp.tmp ex4_4.tmp
    not ok diff-sys_classes_viewer_tests-ex4_4
/Users/patrick/petsc-master/bin/petscdiff
 /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out
ex4_4.tmp > diff-runex4_4.out 2> diff-runex4_4.out
    # 2a3,14
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1
    # > Testing PetscViewerASCIIPrintf 0
    # > Testing PetscViewerASCIIPrintf 1

I go to take a look at the output of the test to see if I can run the
actual executable myself.

  cd
arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests

runex4_4.sh in that directory isn't so informative as far as the command I
should run. It does tell me where I might look for reference output, though


 /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out

 If I look in the runex4_4 directory, I find
sys_classes_viewer_tests-ex4_4.sh which gives me something to copy and
paste and use my actual knowledge/reasoning to fix:

   /Users/patrick/petsc-master/arch-darwin-master-double-debug/bin/mpiexec
 -n 1 ../ex4 -myviewer ascii:ex4a1.tmp::append

(I'm not going to try to debug this failure further, but I'm confident I'd
be able to given the information I have now)

This is pretty close to what I want, which is a way to "back out of the
test harness" to be able to debug the failed test myself. I was hoping that
running this from the PETSc root directory would more directly tell me that:


 arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh
 -v





2018-01-25 5:47 GMT+01:00 Smith, Barry F. <bsmith at mcs.anl.gov>:

>
>    PETSc developers,
>
>      We have completed moving all PETSc examples over from the old test
> system (where tests were written in the makefile) to a new system, provided
> by Scott Kruger, where the test rules are written in bottom of the source
> file of the example. Directions for usage and adding new tests can be found
> in the PETSc developers manual http://www.mcs.anl.gov/petsc/
> petsc-dev/docs/developers.pdf chapter 7.
>
>   Barry
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180125/aab8689f/attachment-0001.html>