[petsc-dev] Test harness upgrade

[Scott Kruger]

As promised, here is the pull request discussed earlier
(before close-of-business Central time!):
  https://bitbucket.org/petsc/petsc/pull-requests/633

To illustrate the point here, I will use examples
from ksp/ksp/examples/tutorials/ex10.c

Consider this test block:

    test:
       suffix: 19
       requires: datafilespath
       args: -f0 ${DATAFILESPATH}/matrices/poisson1
       args: -ksp_type cg -pc_type icc -pc_factor_levels 2
       test:
       test:
          args: -mat_type seqsbaij

which has two subtests or children tests (feel free to rename).
This will generate a shell script runex19.sh that does something like:
        mpiexec -n 1 ex19 ${args} > ex19.temp
        petscdiff ex19.out ex19.tmp
        mpiexec -n 1 ex19 ${args} -mat_type seqsbaij > ex19.temp
        petscdiff ex19.out ex19.tmp


This works because the argument '-mat_type seqsbaij'
DOES NOT CHANGE THE OUTPUT enabling the same output_file
to be used. It also makes for a nice compact specification.

But what if arguments change the output file?
Putting suffix then triggers separate "tests" (defined as
separate shell scripts):

    test:
       requires: datafilespath
       args: -f0 ${DATAFILESPATH}/matrices/medium
       args: -ksp_type bicg
       test:
          suffix: 4
          args: -pc_type lu
       test:
          suffix: 5

This will generate runex4.sh and runex5.sh that only
differ in the '-pc_type lu' in the arguments and in
the output file names.
One can quickly see that they are very similar tests,
and it's very compact in it's representation.


Now consider for loops:
    test:
       suffix: 7_d
       requires: datafilespath
       args: -f0 ${DATAFILESPATH}/matrices/medium
       args: -viewer_binary_skip_info -mat_type seqbaij
       args: -matload_block_size {{2 3 4 5 6 7 8}}
       args: -ksp_type preonly -pc_type lu

This generates:
--------------------------------------
for matload_block_size in 2 3 4 5 6 7 8; do

    petsc_testrun "${mpiexec} -n 1 ${exec} ${args}  -matload_block_siz
e ${matload_block_size}" ex10_7_d.tmp ${testname}.err "${label}_matlo
ad_block_size-${matload_block_size}"

    petsc_testrun "${diff_exe} /Users/kruger/ptsolveall/upstream/petsc
/src/ksp/ksp/examples/tutorials/output/ex10_7_d.out ex10_7_d.tmp" dif
f-${testname}.out diff-${testname}.out diff-${label}_matload_block_si
ze-${matload_block_size} ""

done
--------------------------------------

All 7 diff tests will be against the same file.
Here, I have the reporting being labelled with
the variable name and value to make it easy to
see what is being tested.



What if your for loop creates a different output?

In most of the petsc legacy tests this seemed to be
handled by appending it to one massive file and diffing
the whole thing.  Keeping with the
   1 shell script => 1 file
we want to have it labelled.  Tobin proposed one
naming scheme that gives a lot of developer control,
but I moved to a more automated naming scheme (we'll
see if it is sufficient or not).

Here is an example.  The key variable is
separate_testvars which says that the loop
variable should create separate tests.

    test:
       suffix: 19
       requires: datafilespath
       args: -f0 ${DATAFILESPATH}/matrices/poisson1
       args: -ksp_type cg -pc_type icc
       args: -pc_factor_levels {{0 2 4}}
       separate_testvars: pc_factor_levels
       test:
          args:
       test:
          args: -mat_type seqsbaij

This will generate 3 shell scripts:
   runex10_19_pc_factor_levels-0.sh
   runex10_19_pc_factor_levels-2.sh
   runex10_19_pc_factor_levels-4.sh

Each with a different output file being
tested against (but each shell script
invoking ex19 twice).

Between suffixes and separate_testvars,
it's easy to go nuts (even multiple loops
can make it get very verbose).  Here is
an example from ex10:

    test:
       suffix: 7
       requires: datafilespath
       args: -f0 ${DATAFILESPATH}/matrices/medium
       args: -viewer_binary_skip_info -mat_type seqbaij
       args: -matload_block_size {{2 3 4 5 6 7 8}}
       args: -ksp_max_it 100 -ksp_gmres_cgs_refinement_type refine_always
       args: -ksp_rtol 1.0e-15 -ksp_monitor_short
       separate_testvars: matload_block_size
       test:
          suffix: a
       test:
          suffix: b
          args: -pc_factor_mat_ordering_type nd
       test:
          suffix: c
          args: -pc_factor_levels 1


This creates 21 shell scripts:
runex10_7_matload_block_size-2_a.sh
runex10_7_matload_block_size-2_b.sh
runex10_7_matload_block_size-2_c.sh
runex10_7_matload_block_size-3_a.sh
...


ex10 is a good example because there were
63 runex10* targets in the makefile.  After
conversion, we have 110 shell scripts being
generated coming from this separation of files
instead of concatenation.

The mumps examples are also interesting because
they exercise a lot of mumps and petsc options, but they
are all diffed against the same file.  This uses
the output_file parameter to enable specification
of this same output_file.


Other misc:
testparse.py gives better much better error reporting
although I highly recommend invoking it directly
during development to catch errors.  It saved me
a bunch of time while converting ex10.c.

gmakegentest.py is cleaner.  Many thanks to
Tobin for help in this.

-- 
Tech-X Corporation               kruger at txcorp.com
5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
Boulder, CO 80303                Fax:   (303) 448-7756