[petsc-dev] alternatives to alt files

[Karl Rupp]

>>>        Using alt files for testing is painful. Whenever you add, for example, a new variable to be output in a viewer it changes the output files and you need to regenerate the alt files for all the test configurations. Even though the run behavior of the code hasn't changed.
>>>
>>>       I'm looking for suggestions on how to handle this kind of alternative output in a nicer way (alternative output usually comes from different iterations counts due to different precision and often even different compilers).
>>>
>>>       I idea I was thinking of was instead of having "alt" files we have "patch" files that continue just the patch to the original output file instead of a complete copy. Thus in some situations the patch file would still apply even if the original output file changed thus requiring much less manual work in updating alt files. Essentially the test harness would test against the output file, if that fails it would apply the first patch and compare again, try the second patch etc.
>>
>> yes, a 'patch' approach would simplify updates to the reference
>>
>> However: I'm not sure whether we're tackling the right problem here. Our
>> diff-based testing isn't great. I'd favor more dedicated unit tests,
>> where the correctness check is embedded in the test (ex*.*}) itself
>> rather than determined by some text-based diff tool (which, to make
>> matters worse, even filters out floating point numbers...). Not all
>> tests can be written as such -- but many can and this would
>> significantly reduce the burden on alt files.
> 
> I agree, but I don't know how to change without it being a ton of work.
> We have a huge amount of integration tests, which are basically
> tutorials run in a particular way with output that seemed sensible to
> the developer at the time.  There are a number of unit tests, but my
> impression is that well over half of new tests developed in the past few
> years are integration tests.  It would be better for us to have more
> actual unit tests so that we'd be less reliant on the relatively
> arbitrary convergence characteristics of the integration tests.

yes, moving to more unit tests is a lot of work. At the same time, we 
need a long-term direction; if we just make it easier to do more 
integration tests (what this thread is about) without added incentives 
for unit tests, we should not be surprised if we get even more 
integration tests instead of unit tests in the future ;-)

Best regards,
Karli