[petsc-dev] new test harness in PETSc

[Smith, Barry F.]

  Patrick

    This is very good feed back and understanding. Since you are discovering all this partially documented material you can add the work flows as discussed below to the developers guide (and put your name on the guide) so the future developers don't need to guess and extrapolate as you have. Better to have a little too much in the developers guide than not enough.

   Barry


> On Jan 25, 2018, at 5:35 AM, Patrick Sanan <patrick.sanan at gmail.com> wrote:
> 
> Awesome!
> 
> As a contributor, what I am I supposed to do when a test fails due to some error of mine?
> What's the procedure to be able to run a test "under my own control" (be able to see what's actually being executed, and what's failing in the diff) so that I can fix my error? 
> That is, I want to be able to "back out of the test harness" to be able to debug a failure myself.
> 
> This following is what I would try, given my limited knowledge of how things work and after a quick scan of the docs. Is this as intended or is there an easier way?
> 
> --------------------------------------------------
> 
> I make some seemingly-innocuous changes to the code. (In this case I literally change nothing in master.)
> 
> I want to run all the tests to make sure I didn't break something that I don't understand, so I look at the User Manual and copy and run this command: 
> 
>    make -f gmakefile.test test
> 
> I go away for a while and come back and see that some (3) tests have failed, and that the harness gives me instructions on how to re-run them. 
> 
>     # -------------
>     #   Summary
>     # -------------
>     # FAILED ts_tutorials-ex11_adv_2d_quad_0 diff-sys_classes_viewer_tests-ex4_4 ts_tutorials-ex11_adv_2d_quad_1
>     # success 3051/3915 tests (77.9%)
>     # failed 3/3915 tests (0.1%)
>     # todo 91/3915 tests (2.3%)
>     # skip 770/3915 tests (19.7%)
>     #
>     # To rerun failed tests:
>     #     /opt/local/bin/gmake -f gmakefile test search='ts_tutorials-runex11_adv_2d_quad_0 sys_classes_viewer_tests-runex4_4 ts_tutorials-runex11_adv_2d_quad'
> 
> I want to take a look at the output so I can see what's happening - I see that I'm given instructions on how to get more verbose output, so I run one of the tests again with V=1 : 
> 
>     /opt/local/bin/gmake V=1 -f gmakefile test search='ts_tutorials-runex11_adv_2d_quad_0'
> 
> This tells me that this is failing because of a timeout
> 
>     #	Exceeded timeout limit of 60 s
> 
> I poke around in the dev manual (note that "make -f gmakefile help" and "make -f gmakefile.test help" say nothing about TIMEOUT as promised)
> and eventually run
> 
>      /opt/local/bin/gmake V=1 TIMEOUT=1000 -f gmakefile test search='ts_tutorials-runex11_adv_2d_quad_0'
> 
> and get success. Sweet - my assumption that this failed because I'm running a debug build on a slow laptop is confirmed.
> 
>     # -------------
>     #   Summary
>     # -------------
>    # success 2/2 tests (100.0%)
>     # failed 0/2 tests (0.0%)
>     # todo 0/2 tests (0.0%)
>     # skip 0/2 tests (0.0%)
> 
> I then look at the other failed test. 
> 
>      /opt/local/bin/gmake V=1 -f gmakefile test search='sys_classes_viewer_tests-runex4_4'
> 
> This one seems to be failing because of the diff:
> 
>     arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh  -v
>     ok sys_classes_viewer_tests-ex4_4 cat ex4_4.tmp | > /dev/null; cat ex4a1.tmp > ex4_4.tmp.tmp 2>> runex4_4.err && mv ex4_4.tmp.tmp ex4_4.tmp
>     not ok diff-sys_classes_viewer_tests-ex4_4 /Users/patrick/petsc-master/bin/petscdiff  /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out ex4_4.tmp > diff-runex4_4.out 2> diff-runex4_4.out
>     #	2a3,14
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
>     #	> Testing PetscViewerASCIIPrintf 0
>     #	> Testing PetscViewerASCIIPrintf 1
> 
> I go to take a look at the output of the test to see if I can run the actual executable myself. 
> 
>   cd arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests
> 
> runex4_4.sh in that directory isn't so informative as far as the command I should run. It does tell me where I might look for reference output, though
> 
>      /Users/patrick/petsc-master/src/sys/classes/viewer/examples/tests/output/ex4a.out
> 
>  If I look in the runex4_4 directory, I find sys_classes_viewer_tests-ex4_4.sh which gives me something to copy and paste and use my actual knowledge/reasoning to fix:
> 
>    /Users/patrick/petsc-master/arch-darwin-master-double-debug/bin/mpiexec  -n 1 ../ex4 -myviewer ascii:ex4a1.tmp::append
> 
> (I'm not going to try to debug this failure further, but I'm confident I'd be able to given the information I have now)
> 
> This is pretty close to what I want, which is a way to "back out of the test harness" to be able to debug the failed test myself. I was hoping that running this from the PETSc root directory would more directly tell me that:
> 
>    arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh  -v
> 
> 
> 
> 
> 
> 2018-01-25 5:47 GMT+01:00 Smith, Barry F. <bsmith at mcs.anl.gov>:
> 
>    PETSc developers,
> 
>      We have completed moving all PETSc examples over from the old test system (where tests were written in the makefile) to a new system, provided by Scott Kruger, where the test rules are written in bottom of the source file of the example. Directions for usage and adding new tests can be found in the PETSc developers manual http://www.mcs.anl.gov/petsc/petsc-dev/docs/developers.pdf chapter 7.
> 
>   Barry
> 
> 
>