[petsc-dev] Test harness upgrade

[Barry Smith]

> On Feb 10, 2017, at 5:18 PM, Scott Kruger <kruger at txcorp.com> wrote:
> 
> 
> As promised, here is the pull request discussed earlier
> (before close-of-business Central time!):
> https://bitbucket.org/petsc/petsc/pull-requests/633
> 
> To illustrate the point here, I will use examples
> from ksp/ksp/examples/tutorials/ex10.c
> 
> Consider this test block:
> 
>   test:
>      suffix: 19
>      requires: datafilespath
>      args: -f0 ${DATAFILESPATH}/matrices/poisson1
>      args: -ksp_type cg -pc_type icc -pc_factor_levels 2
>      test:
>      test:

  Why is test listed twice here?

>         args: -mat_type seqsbaij
> 
> which has two subtests or children tests (feel free to rename).
> This will generate a shell script runex19.sh that does something like:
>       mpiexec -n 1 ex19 ${args} > ex19.temp
>       petscdiff ex19.out ex19.tmp
>       mpiexec -n 1 ex19 ${args} -mat_type seqsbaij > ex19.temp
>       petscdiff ex19.out ex19.tmp
> 
> 
> This works because the argument '-mat_type seqsbaij'
> DOES NOT CHANGE THE OUTPUT enabling the same output_file
> to be used. It also makes for a nice compact specification.
> 
> But what if arguments change the output file?
> Putting suffix then triggers separate "tests" (defined as
> separate shell scripts):
> 
>   test:
>      requires: datafilespath
>      args: -f0 ${DATAFILESPATH}/matrices/medium
>      args: -ksp_type bicg
>      test:
>         suffix: 4
>         args: -pc_type lu
>      test:
>         suffix: 5
> 
> This will generate runex4.sh and runex5.sh that only
> differ in the '-pc_type lu' in the arguments and in
> the output file names.
> One can quickly see that they are very similar tests,
> and it's very compact in it's representation.

  So the indentation is what indicates it is a subtest versus a completely new test?
> 
> 
> Now consider for loops:
>   test:
>      suffix: 7_d
>      requires: datafilespath
>      args: -f0 ${DATAFILESPATH}/matrices/medium
>      args: -viewer_binary_skip_info -mat_type seqbaij
>      args: -matload_block_size {{2 3 4 5 6 7 8}}
>      args: -ksp_type preonly -pc_type lu
> 
> This generates:
> --------------------------------------
> for matload_block_size in 2 3 4 5 6 7 8; do
> 
>   petsc_testrun "${mpiexec} -n 1 ${exec} ${args}  -matload_block_siz
> e ${matload_block_size}" ex10_7_d.tmp ${testname}.err "${label}_matlo
> ad_block_size-${matload_block_size}"
> 
>   petsc_testrun "${diff_exe} /Users/kruger/ptsolveall/upstream/petsc
> /src/ksp/ksp/examples/tutorials/output/ex10_7_d.out ex10_7_d.tmp" dif
> f-${testname}.out diff-${testname}.out diff-${label}_matload_block_si
> ze-${matload_block_size} ""
> 
> done
> --------------------------------------
> 
> All 7 diff tests will be against the same file.
> Here, I have the reporting being labelled with
> the variable name and value to make it easy to
> see what is being tested.
> 
> 
> 
> What if your for loop creates a different output?
> 
> In most of the petsc legacy tests this seemed to be
> handled by appending it to one massive file and diffing
> the whole thing.  Keeping with the
>  1 shell script => 1 file
> we want to have it labelled.  Tobin proposed one
> naming scheme that gives a lot of developer control,
> but I moved to a more automated naming scheme (we'll
> see if it is sufficient or not).
> 
> Here is an example.  The key variable is
> separate_testvars which says that the loop
> variable should create separate tests.
> 
>   test:
>      suffix: 19
>      requires: datafilespath
>      args: -f0 ${DATAFILESPATH}/matrices/poisson1
>      args: -ksp_type cg -pc_type icc
>      args: -pc_factor_levels {{0 2 4}}
>      separate_testvars: pc_factor_levels

   Without the - ? 
>      test:
>         args:
>      test:
>         args: -mat_type seqsbaij
> 

   Instead of having the ugly separate_testvars argument could you somehow indicate it with the loop definition? 

> args: -pc_factor_levels {{0 2 4}separate test output files} 

and 

> args: -pc_factor_levels {{0 2 4}shared test output file} 



> This will generate 3 shell scripts:
>  runex10_19_pc_factor_levels-0.sh
>  runex10_19_pc_factor_levels-2.sh
>  runex10_19_pc_factor_levels-4.sh
> 
> Each with a different output file being
> tested against (but each shell script
> invoking ex19 twice).
> 
> Between suffixes and separate_testvars,
> it's easy to go nuts (even multiple loops
> can make it get very verbose).  Here is
> an example from ex10:
> 
>   test:
>      suffix: 7
>      requires: datafilespath
>      args: -f0 ${DATAFILESPATH}/matrices/medium
>      args: -viewer_binary_skip_info -mat_type seqbaij
>      args: -matload_block_size {{2 3 4 5 6 7 8}}
>      args: -ksp_max_it 100 -ksp_gmres_cgs_refinement_type refine_always
>      args: -ksp_rtol 1.0e-15 -ksp_monitor_short
>      separate_testvars: matload_block_size
>      test:
>         suffix: a
>      test:
>         suffix: b
>         args: -pc_factor_mat_ordering_type nd
>      test:
>         suffix: c
>         args: -pc_factor_levels 1
> 
> 
> This creates 21 shell scripts:
> runex10_7_matload_block_size-2_a.sh
> runex10_7_matload_block_size-2_b.sh
> runex10_7_matload_block_size-2_c.sh
> runex10_7_matload_block_size-3_a.sh
> ...
> 
> 
> ex10 is a good example because there were
> 63 runex10* targets in the makefile.  After
> conversion, we have 110 shell scripts being
> generated coming from this separation of files
> instead of concatenation.
> 
> The mumps examples are also interesting because
> they exercise a lot of mumps and petsc options, but they
> are all diffed against the same file.  This uses
> the output_file parameter to enable specification
> of this same output_file.
> 
> 
> Other misc:
> testparse.py gives better much better error reporting
> although I highly recommend invoking it directly
> during development to catch errors.  It saved me
> a bunch of time while converting ex10.c.
> 
> gmakegentest.py is cleaner.  Many thanks to
> Tobin for help in this.
> 
> -- 
> Tech-X Corporation               kruger at txcorp.com
> 5621 Arapahoe Ave, Suite A       Phone: (720) 974-1841
> Boulder, CO 80303                Fax:   (303) 448-7756