<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">
<div class="moz-cite-prefix"><font size="-1"><b>On 11/21/2017
02:33 AM, Wei-keng Liao wrote:</b></font><br>
</div>
<blockquote type="cite"
cite="mid:3B8CD655-4D16-475F-8900-60C2AA88B2A7@eecs.northwestern.edu">
<blockquote>
<pre wrap="">The "make check" error reporting mechanism has been changed after migrating to automake.
It is now using the script-based style of automake. See automake guide in
<a class="moz-txt-link-freetext" href="http://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html">http://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html</a>
After running "make check", the outputs (to stdout and stderr) of test programs are stored
in test/*/seq_runs.sh.log. From there you will see those familiar outputs from 1.8.1.
In the new design, the outputs of all test programs in the same folder will go to the
same log file.
Running "make check" will stop immediately when encountering the first failed test and
print the failure messages on screen. An example of failed make check with error messages
is given below. If you did not see any error/failed message at the end, then all tests
were passed. To make that more obvious, I will add a print statement at the end of
"make" and "make check" to indicate whether libraries have been built and all test programs
have passed, respectively.
</pre>
</blockquote>
</blockquote>
I always use<br>
<blockquote><tt>make -i -k check</tt><br>
</blockquote>
to force all the tests to run. This way I get a full accounting of
what worked and what didn't.<br>
There have been cases with app's where I've been assured that some
tests don't matter for various reasons, so I want to see if these
are the only ones failing, or if the failures went away with a the
latest version of the source base.<br>
As far as the tool goes, I get this kind of output from NetCDF-C
for example:<tt><br>
</tt>
<blockquote><tt>PASS: tst_h_strings2</tt><br>
<tt>PASS: tst_h_ints</tt><br>
<tt>FAIL: tst_h_dimscales</tt><br>
<tt>PASS: tst_h_dimscales1</tt><br>
<tt>PASS: tst_h_dimscales2</tt><br>
<tt>PASS: tst_h_dimscales3</tt><br>
<tt>PASS: tst_h_enums</tt><br>
<tt>PASS: tst_h_dimscales4</tt><br>
<tt>PASS: run_par_tests.sh</tt><br>
<tt>============================================================================</tt><br>
<tt>Testsuite summary for netCDF 4.5.0</tt><br>
<tt>============================================================================</tt><br>
<tt># TOTAL: 27</tt><br>
<tt># PASS: 26</tt><br>
<tt># SKIP: 0</tt><br>
<tt># XFAIL: 0</tt><br>
<tt># FAIL: 1</tt><br>
<tt># XPASS: 0</tt><br>
<tt># ERROR: 0</tt></blockquote>
so while it's running a battery of sub-tests here, it's giving me
counts of them individually rather than just a total of one pass.<br>
Are you using the same toolkit that they are?<br>
<br>
-- Carl<br>
<br>
<hr size="2" width="100%"><font size="-1"><b>On 11/21/2017 02:33
AM, Wei-keng Liao wrote:
</b></font></div>
<blockquote type="cite"
cite="mid:3B8CD655-4D16-475F-8900-60C2AA88B2A7@eecs.northwestern.edu">
<pre wrap="">
Example of screen shot for failed "make check":
...
Making check in test
Making check in common
Making check in C
PASS: seq_runs.sh
============================================================================
Testsuite summary for parallel-netcdf 1.9.0.pre1
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
Making check in fandc
Making check in nc_test
FAIL: seq_runs.sh
============================================================================
Testsuite summary for parallel-netcdf 1.9.0.pre1
============================================================================
# TOTAL: 1
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See test/nc_test/test-suite.log
Please report to <a class="moz-txt-link-abbreviated" href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>
============================================================================
make[5]: *** [test-suite.log] Error 1
make[4]: *** [check-TESTS] Error 2
make[3]: *** [check-am] Error 2
make[2]: *** [check] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check-recursive] Error 1
</pre>
</blockquote>
<hr size="2" width="100%">
<pre wrap=""><b>On Nov 21, 2017, at 1:38 AM, Carl Ponder wrote:</b>
</pre>
<blockquote type="cite"
cite="mid:3B8CD655-4D16-475F-8900-60C2AA88B2A7@eecs.northwestern.edu">
<pre wrap="">A question on the regression-tests: in this pre-release version, I see 10 regression-tests being reported
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
# TOTAL: 0
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
# TOTAL: 1
whereas in 1.8.1, there were 86 tests reported:
*** TESTING C nc_test for format CDF-1
*** TESTING C nc_test for format CDF-2
*** TESTING C nc_test for format CDF-5
*** TESTING C t_nc for emulating netCDF t_nc ------ pass
*** TESTING C tst_misc for emulating netCDF t_misc ------ pass
*** TESTING C tst_norm for emulating netCDF tst_norm ------ pass
*** TESTING C tst_small for emulating netCDF tst_small ------ pass
*** TESTING C tst_names for emulating netCDF tst_names ------ pass
*** TESTING C tst_atts3 for emulating netCDF tst_atts3 ------ pass
*** TESTING C tst_atts for emulating netCDF tst_atts ------ pass
*** TESTING C tst_nofill for fill/nofill modes ------ pass
*** TESTING C pres_temp_4D_wr for writing file ------ pass
*** TESTING C pres_temp_4D_rd for reading file ------ pass
*** TESTING C mcoll_perf for mput/iput APIs ------ pass
*** TESTING C test_bput for bput API ------ pass
*** TESTING C interleaved for writing interleaved fileviews ------ pass
*** TESTING C i_varn_int64 for iput/iget varn ------ pass
*** TESTING C flexible_bput for flexible bput_varm ------ pass
*** TESTING C wait_after_indep for ncmpi_end_indep_data ------ pass
*** TESTING C req_all for NC_REQ_ALL ------ pass
*** TESTING C i_varn_indef for iput/iget varn in define mode ------ pass
*** TESTING C bput_varn for bput_varn ------ pass
*** TESTING C column_wise for iput/iget interleaved access ------ pass
*** TESTING F77 ./mcoll_testf77 for iput API ------ pass
*** TESTING F77 ./test_bputf77 for bput_varm_real API ------ pass
*** TESTING F90 ./mcoll_testf for nf90mpi_iput_var API ------ pass
*** TESTING F90 ./test_bputf for bput_var ------ pass
*** TESTING C test_inq_format for inquiring CDF file formats ------ pass
*** TESTING C cdf_type for CDF-5 type in CDF-1 and 2 ------ pass
*** TESTING C dim_cdf12 for defining dim in CDF-1/2 format ------ pass
*** TESTING C ncmpi_vars_null_stride for NULL stride ------ pass
*** TESTING C vectors for put_vara/get_vara ------ pass
*** TESTING C collective_error for collective abort ------ pass
*** TESTING C test_varm for get/put varm ------ pass
*** TESTING C alignment_test for alignment ------ pass
*** TESTING C flexible for flexible put and get ------ pass
*** TESTING C flexible2 for flexible APIs ------ pass
*** TESTING C flexible_varm for flexible varm APIs ------ pass
*** TESTING C nonblocking for using ncmpi_iput_vara_int() ------ pass
*** TESTING C noclobber for NC_NOCLOBBER and NC_EEXIST ------ pass
*** TESTING C record for write records in reversed order ------ pass
*** TESTING C inq_num_vars for no. record/fixed variables ------ pass
*** TESTING C varn_int for ncmpi_put_varn_int_all() ------ pass
*** TESTING C modes for file create/open modes
*** TESTING C one_record for only one record variable ------ pass
*** TESTING C inq_recsize for inquiring record size ------ pass
*** TESTING C test_vard for flexible put and get ------ pass
*** TESTING C varn_contig for put_varn with contig fileview ------ pass
*** TESTING C ivarn for ncmpi_iput_varn_<type>() ------ pass
*** TESTING C check_striping for strining info ------ pass
*** TESTING C add_var for checking offsets of new variables ------ pass
*** TESTING C buftype_free for free buftype in flexible API ------ pass
*** TESTING C last_large_var for last large var in CDF-1/2 ------ pass
*** TESTING C check_type for checking for type conflict ------ pass
*** TESTING C test_erange for checking for NC_ERANGE ------ pass
*** TESTING C scalar for get/put scalar variables ------ pass
*** TESTING C redef1 for entering re-define mode ------ pass
*** TESTING F77 ./varn_intf for varn API ------ pass
*** TESTING F77 ./attrf for attribute overflow ------ pass
*** TESTING F77 ./buftype_freef for flexible API ------ pass
*** TESTING F77 ./put_parameter for using immutable write buf ------ pass
*** TESTING F77 ./test_vardf for vard API ------ pass
*** TESTING F90 ./inq_num_varsf for no. record/fixed variables ------ pass
*** TESTING F90 ./inq_recsizef for inquiring record size ------ pass
*** TESTING F90 ./test_vardf90 for vard API ------ pass
*** TESTING F90 ./varn_real for varn API ------ pass
*** TESTING C erange_fill for checking for type conflict ------ pass
*** TESTING C redef1 for entering re-define mode ------ pass
*** TESTING C++ nctst for APIs with different netCDF formats ------ pass
*** TESTING C++ test_classic for creation of classic format file ------ pass
*** TESTING F77 ./nf_test for CDF-1 ------ pass
*** TESTING F77 ./nf_test for CDF-2 ------ pass
*** TESTING F77 ./nf_test for CDF-5 ------ pass
*** TESTING F90 ./nf90_test for CDF-1 ------ pass
*** TESTING F90 ./nf90_test for CDF-2 ------ pass
*** TESTING F90 ./nf90_test for CDF-5 ------ pass
*** TESTING F90 ./test_intent for INTENT modifier ------ pass
*** TESTING F90 ./tst_f90 ------ pass
*** TESTING F90 ./f90tst_vars for def_var API ------ pass
*** TESTING F90 ./tst_types2 for 64-bit integer types ------ pass
*** TESTING F90 ./tst_f90_cdf5 ------ pass
*** TESTING F90 ./f90tst_vars2 for def_var API ------ pass
*** TESTING F90 ./f90tst_vars3 for def_var API ------ pass
*** TESTING F90 ./f90tst_vars4 for def_var API ------ pass
*** TESTING F90 ./tst_flarge for large files ------ pass
*** TESTING F90 ./tst_io ------ pass
In the 1.9.0 list, are you aggregating the the results of more than one test in the 1-total's?
For example, it looks like you're compiling several tests here but running a single driver, and reporting it as a single test:
Making check in cdf_format
make[2]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
make test_inq_format cdf_type dim_cdf12 tst_open_cdf5 tst_corrupt
make[3]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
CC test_inq_format.o
CCLD test_inq_format
CC cdf_type.o
CCLD cdf_type
CC dim_cdf12.o
CCLD dim_cdf12
CC tst_open_cdf5.o
CCLD tst_open_cdf5
CC tst_corrupt.o
CCLD tst_corrupt
make[3]: Leaving directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
make check-TESTS
make[3]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
make[4]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
PASS: seq_runs.sh
============================================================================
Testsuite summary for parallel-netcdf 1.9.0.pre1
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
This would make it difficult to identify any single failures.
</pre>
</blockquote>
<DIV>
<HR>
</DIV>
<DIV>This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original
message. </DIV>
<DIV>
<HR>
</DIV>
</body>
</html>