[petsc-dev] alternatives to alt files

[Hapla Vaclav]

On 3 May 2019, at 06:21, Karl Rupp via petsc-dev <petsc-dev at mcs.anl.gov<mailto:petsc-dev at mcs.anl.gov>> wrote:

Hi,

   Scott and PETSc folks,
     Using alt files for testing is painful. Whenever you add, for example, a new variable to be output in a viewer it changes the output files and you need to regenerate the alt files for all the test configurations. Even though the run behavior of the code hasn't changed.
    I'm looking for suggestions on how to handle this kind of alternative output in a nicer way (alternative output usually comes from different iterations counts due to different precision and often even different compilers).
    I idea I was thinking of was instead of having "alt" files we have "patch" files that continue just the patch to the original output file instead of a complete copy. Thus in some situations the patch file would still apply even if the original output file changed thus requiring much less manual work in updating alt files. Essentially the test harness would test against the output file, if that fails it would apply the first patch and compare again, try the second patch etc.

yes, a 'patch' approach would simplify updates to the reference

However: I'm not sure whether we're tackling the right problem here. Our diff-based testing isn't great. I'd favor more dedicated unit tests, where the correctness check is embedded in the test (ex*.*}) itself rather than determined by some text-based diff tool (which, to make matters worse, even filters out floating point numbers...). Not all tests can be written as such -- but many can and this would significantly reduce the burden on alt files.

Hi Karli,

I'm not sure you about this nomenclature. Let me take DMPlexInterpolate as an example:
So (1) comparing the actual view of the interpolated mesh to the snapshot stored in the output file would be an integration test, whereas (2) calling a set of DMPlexCheck* functions, which would fail if something is wrong in the produced topology, would be a unit test?

I wouldn't use the term integration testing here because here I would test exactly the same level of functionality, whereas integration testing should mean one level up, i.e. tests how parts of the system work together<https://codeutopia.net/blog/2015/04/11/what-are-unit-testing-integration-testing-and-functional-testing/>. Or do you mean those checks could be run as separate tests while the diff test is indivisible? But these checks could be for example easily turned into functions returning booleans, and that test would print these booleans and compare with the output file. I think this wouldn't change it from a unit test to an integration test.

What PETSc diff tests use could be IMHO called snapshot/golden master/characterization testing [approach], applied to both unit and integration testing [levels].
https://sqa.stackexchange.com/questions/29696/what-is-snapshot-testing
https://lists.boost.org/boost-users/2013/04/78334.php

I really don't want to be knit-picking here; I just to want to clearly understand what is the intention.

Best,
Vaclav

BTW https://en.wikipedia.org/wiki/Characterization_test hurts a bit:
- "protect existing behavior of legacy code against unintended changes"
- "provide a safety net for extending and refactoring code that does not have adequate unit tests"
- "this kind of testing corresponds to the historical oracle"
and references like
- Surviving Legacy Code with Golden Master and Sampling
- Working Effectively with Legacy Code
:-D
So let's rather call it snapshot testing ;-)


Best regards,
Karli


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190503/e480fcf6/attachment.html>