[petsc-dev] Nightly tests quick summary page

[Karl Rupp]

Hi Jed,

 >     It's also a matter of the need for a 'yes/no'-testing. Running a
>     fixed test like
>       if (err > eps) fail_test();
>     is probably too harsh and we instead use some kind of continuous
>     metric to judge the outcome of the test.
>
>
> Agreed, but note that an arbitrarily small continuum difference can
> force an additional iteration. Maybe our tests should be configured to
> "always run exactly 10 iterations". I wonder if a Bayesian filter could
> be trained to recognize real failures.

Testing for the same number of iterations is - as you mentioned - a 
terrible metric. I see this regularly on GPUs, where rounding modes 
differ slightly from CPUs. Running a fixed (low) number of iterations is 
certainly the better choice here, provided that the systems we use for 
the tests are neither too ill-conditioned nor too well-behaved so that 
we can eventually reuse the tests for some preconditioners.


>     Speaking of a HTML table, something that is spinning in my head for
>     a long time already is that one can easily draw diagrams
>     automatically showing the convergence history of the residual norm
>     obtained in a test run. Coloring the frame of the plot proportional
>     to the relative deviation from a reference convergence history gives
>     you a quick idea of how far a test is off the reference. It won't
>     work for all tests, but it gives you on idea about the sanity of the
>     implementation.
>
>
> I spend years trying to convince people to use actual norms instead of
> jpeg semi-norms and now you want to reverse that? Blasphemy!

Haha :-D Did I just throw the first stone? ;-)

> Actually, that's an interesting idea. I guess it could be presented in a
> more compressed way than the numerical values, thus providing a better
> summary.

Yep. Actual norms should still be used for collecting 'hard facts' about 
the state, and a 'pretty picture layer' will drive up efficiency in 
interpreting the results.

Best regards,
Karli