[petsc-dev] new test harness in PETSc

[Scott Kruger]

On 1/25/18 4:35 AM, Patrick Sanan wrote:
> 
> This following is what I would try, given my limited knowledge of how 
> things work and after a quick scan of the docs. Is this as intended or 
> is there an easier way?
You pretty much have it.

> 
> --------------------------------------------------
> 
> I make some seemingly-innocuous changes to the code. (In this case I 
> literally change nothing in master.)
> 
> I want to run all the tests to make sure I didn't break something that I 
> don't understand, so I look at the User Manual and copy and run this 
> command:
> 
>     make -f gmakefile.test test

This will be hooked up to the main makefile test target at some point.
Right now,
make allgtest
is more silent for example, but you can pick out the errors.
make allgtest-tap
is equivalent to above (you see progress, but you have to pick out
the errrors that occurred in 0.1% of tests).

> 
> I go away for a while and come back and see that some (3) tests have 
> failed, and that the harness gives me instructions on how to re-run them.
> 
>      # -------------
>      #   Summary
>      # -------------
>      # FAILED ts_tutorials-ex11_adv_2d_quad_0 
> diff-sys_classes_viewer_tests-ex4_4 ts_tutorials-ex11_adv_2d_quad_1
>      # success 3051/3915 tests (77.9%)
>      # failed 3/3915 tests (0.1%)
>      # todo 91/3915 tests (2.3%)
>      # skip 770/3915 tests (19.7%)
>      #
>      # To rerun failed tests:
>      #     /opt/local/bin/gmake -f gmakefile test 
> search='ts_tutorials-runex11_adv_2d_quad_0 
> sys_classes_viewer_tests-runex4_4 ts_tutorials-runex11_adv_2d_quad'
> 
> I want to take a look at the output so I can see what's happening - I 
> see that I'm given instructions on how to get more verbose output, so I 
> run one of the tests again with V=1 :
> 
>      /opt/local/bin/gmake V=1 -f gmakefile test 
> search='ts_tutorials-runex11_adv_2d_quad_0'
> 
> This tells me that this is failing because of a timeout
> 
>      #Exceeded timeout limit of 60 s
> 
> I poke around in the dev manual (note that "make -f gmakefile help" and 
> "make -f gmakefile.test help" say nothing about TIMEOUT as promised)

Oops.  Yes, a bug.



>   If I look in the runex4_4 directory, I find 
> sys_classes_viewer_tests-ex4_4.sh which gives me something to copy and 
> paste and use my actual knowledge/reasoning to fix:
> 
>    
>   /Users/patrick/petsc-master/arch-darwin-master-double-debug/bin/mpiexec  -n 1 ../ex4 -myviewer ascii:ex4a1.tmp::append
> 
> (I'm not going to try to debug this failure further, but I'm confident 
> I'd be able to given the information I have now)
> 
> This is pretty close to what I want, which is a way to "back out of the 
> test harness" to be able to debug the failed test myself. I was hoping 
> that running this from the PETSc root directory would more directly tell 
> me that:
> 
>    
>   arch-darwin-master-double-debug/tests/src/sys/classes/viewer/examples/tests/runex4_4.sh  -v


The output is meant to be user facing (in which case they
do not want to be told how to debug -- they just want to
report it), and to be run for nightlies -- in which case
we want quite a bit of terseness.

I think the fact that you figured it out on your own
is a success!  You did the workflow I use and yes, the
first time is annoying, but then it's pretty straightforward.

A few comments:
The additional step of poking around the runex4_4 directory
is to avoid output file name clashes.

The  runex4_4.sh script has a -h command to show other options,
especially the ability to automatically update the
output files.

NO_RM=1 is my favorite option to gmakefile.test, but
I have shared builds.  Somewhere we have a change that turns this on
if shared builds are there so debugging is faster.

Scott



> 
> 
> 
> 
> 
> 2018-01-25 5:47 GMT+01:00 Smith, Barry F. <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>>:
> 
> 
>         PETSc developers,
> 
>           We have completed moving all PETSc examples over from the old
>     test system (where tests were written in the makefile) to a new
>     system, provided by Scott Kruger, where the test rules are written
>     in bottom of the source file of the example. Directions for usage
>     and adding new tests can be found in the PETSc developers manual
>     http://www.mcs.anl.gov/petsc/petsc-dev/docs/developers.pdf
>     <http://www.mcs.anl.gov/petsc/petsc-dev/docs/developers.pdf> chapter 7.
> 
>        Barry
> 
> 
> 

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756