[petsc-dev] alternatives to alt files
jed at jedbrown.org
Fri May 3 10:20:28 CDT 2019
A recent post comparing performance of some of those unit testing frameworks.
My point about integration tests is that if we had more unit tests, the
integration tests could have less reliance on long-form convergence
logs, thus reducing the number of tests of the kind where alt files are
most often needed. I'd argue it isn't orthogonal, but still
independent: unit testing won't eliminate such cases.
"Smith, Barry F. via petsc-dev" <petsc-dev at mcs.anl.gov> writes:
> Most packages that have good coverage with unit tests use a unit test system to manage the unit test code. I was actually surprised how many there are for C https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C. But selecting evaluating and implementing a unit test environment for PETSc is IMHO orthogonal to my question. The current question is reqarding our current integration test environment and improving it incrementally (another orthogonal incremental improvement might be adding the ability to do diffs on floating point numbers in a coherent way). It would be wrong in IMHO to say we shouldn't improve the current integration tests because it makes developing a unit test system less likely. We'll always need a very robust integration tests regardless of how good our future unit test system is so no work is lost.
> I'd be happy to see a pull request with a unit test system, but orthogonal ;)
>> On May 2, 2019, at 10:21 PM, Karl Rupp <rupp at iue.tuwien.ac.at> wrote:
>>> Scott and PETSc folks,
>>> Using alt files for testing is painful. Whenever you add, for example, a new variable to be output in a viewer it changes the output files and you need to regenerate the alt files for all the test configurations. Even though the run behavior of the code hasn't changed.
>>> I'm looking for suggestions on how to handle this kind of alternative output in a nicer way (alternative output usually comes from different iterations counts due to different precision and often even different compilers).
>>> I idea I was thinking of was instead of having "alt" files we have "patch" files that continue just the patch to the original output file instead of a complete copy. Thus in some situations the patch file would still apply even if the original output file changed thus requiring much less manual work in updating alt files. Essentially the test harness would test against the output file, if that fails it would apply the first patch and compare again, try the second patch etc.
>> yes, a 'patch' approach would simplify updates to the reference
>> However: I'm not sure whether we're tackling the right problem here. Our diff-based testing isn't great. I'd favor more dedicated unit tests, where the correctness check is embedded in the test (ex*.*}) itself rather than determined by some text-based diff tool (which, to make matters worse, even filters out floating point numbers...). Not all tests can be written as such -- but many can and this would significantly reduce the burden on alt files.
>> Best regards,
More information about the petsc-dev