thoughts on testing and coverage

[Miller, Mark C.]

We discussed coverage metrics in the context of iMesh/iMeshP unit testing.

One useful metric is of course function count. However, many functions can be
called with a variety of arguments including arguments that can drive the interfaces
into error conditions and so function count alone can wind up not providing clear
coverage metric for testing.

We observed that routinely measuring coverage in the same way we routinely
run tests and report results is probably NOT as essential as doing a focused
gcov-like analysis as testing codes are developed and evolve.

With the exception of iMesh reference implementation, existing iMesh/iMeshP
implementations generally include a combination of some other product (e.g.
FMDB, MOAB, GRUMMP) with an ITAPS interface wrapper bolted onto it. To
adequately measure gcov-like coverage if ITAPS interfaces in this setting, we
could take the following appraoch...

For any implementation, we identify the handful of source files were the ITAPS
interface wrapper code exists. Then, that wrapper code is prepared as necessary
for gcov and the gcov analysis is performed ONLY on those relevant files. In this
way, we measure coverage of the ITAPS interfaces WITHOUT having the
whole of the underlying implementation substrate factor into the analysis.

As Jason observed, you'd only need to do this analysis as the test code itself
is developed and modified, but as implementations change.

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/tstt-interface/attachments/20100826/cdc445b8/attachment.htm>