[petsc-dev] bullshit in Plex example under complex

[Barry Smith]

> On Aug 29, 2015, at 6:11 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Barry Smith <bsmith at mcs.anl.gov> writes:
>>  It seems to require a lot of boilerplate for each test. I sure as heck hope we don't have to write that by hand for each test.
> 
> Here is an example:
> 
> $ cat src/hpgmg/finite-element/test/t230-fmg-poisson2.sh 
> #!/bin/sh
> 
> test_description='Test Q2 Poisson solve using FMG'

  Hmm, the example code already says what it does. Repeating it here is annoying

> 
> . ./hpgmg-sharness.sh
> 
> # Error norm is converging at fourth order (superconvergent at Lagrange nodes)

  Again the example should really be providing this information, not test code

> test_expect_stdout 'FE Poisson FMG solve fedegree=2 serial' 1 'hpgmg-fe fmg -op_type poisson2 -M 4,4,6 -smooth 4,3' '
> F(4,3)  0: |e|_2/|u|_2 9.08e-03  |r|_2/|f|_2 3.35e-04
> V(4,3)  1: |e|_2/|u|_2 9.17e-03  |r|_2/|f|_2 8.27e-07
> V(4,3)  2: |e|_2/|u|_2 9.17e-03  |r|_2/|f|_2 5.54e-09

  Input arguments, out expected, fine.

   How are you indicating when what tests can be done. For example requires MUMPS, triangle and double precision. Or runs with all precisions both complex and real. etc etc.


> '
> 
> test_expect_stdout 'FE Poisson FMG solve fedegree=2 parallel' 4 'hpgmg-fe fmg -op_type poisson2 -M 4,4,6 -smooth 4,3 -p 1,2,2' '
> F(4,3)  0: |e|_2/|u|_2 9.08e-03  |r|_2/|f|_2 3.35e-04
> V(4,3)  1: |e|_2/|u|_2 9.17e-03  |r|_2/|f|_2 8.27e-07
> V(4,3)  2: |e|_2/|u|_2 9.17e-03  |r|_2/|f|_2 5.54e-09
> '
> 
> test_done
> 
> 
> Since I have quite short output here, the test output is inline.
> 
>>  I'm not a fan of the "parse the PETSc makefiles approach"; I think we need to define a standard format for declaring tests that is very concise (only exactly what is needed; I have one try in a branch somewhere). Then we can have code that translates from the standard format to anything people want to try with testing scripts. I am even willing to manually convert makefile crap to this format when necessary 
> 
> My "parse the makefiles" was intended to automate the conversion to a
> more concise format that a (hopefully sane) test system can use
> directly.  I'm not wild about introducing extra translation phases
> because it makes the testing system more difficult to understand and
> exposes more possible failure modes.

   Even if we decided to use the sharness format we still need to write a parser so we can process the input files to see what we are testing and what we are not testing, generate tables of functionality tested etc. So I am very much opposed to picking one particular testers format and using that. We need our own format and then SIMPLE tools to translate to various testers. I agree the format must be simple and the tools that translate simple so we are not stuck in a morass of broken tester

> 
> FWIW, we could greatly simplify the makefiles by having each target
> simply call a script that runs the actual program, collects stdout and
> stderr and exit codes, and outputs the test result in a standard format
> (e.g., TAP).  I.e., move the repetitive shell code we have now in each
> target into a single script used for all tests.  This script could also
> have an argument containing restrictions (NoComplex, etc.) so that it
> can determine when to skip.  Moreover, it can sandbox the execution to a
> temporary directory and clean up after it.  As I see it, this is
> basically equivalent to moving the tests into a sharness-type system or
> similar.
> 
> In my opinion, one crucial component in a revamped testing system is to
> enable parallel execution with a maximum number of processes (perhaps
> exceeded when a single job needs more than that, but definitely not
> otherwise).

   Sure. Almost all our tests, in theory, can be run in any order or at the same time.

>  Sharness is really nice for tests that replicate a session
> where the state is tested as various checkpoints along the way.  We have
> a few of these (e.g., creating and reading files), but most of our tests
> are independent, which maps to sharness as independent test scripts.
> One file per test is actually the best case for merging, but can be
> cumbersome otherwise.