regression harness in pre-release of PnetCDF 1.9.0
Wei-keng Liao
wkliao at eecs.northwestern.edu
Tue Nov 21 02:33:15 CST 2017
Hi, Carl
The "make check" error reporting mechanism has been changed after migrating to automake.
It is now using the script-based style of automake. See automake guide in
http://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html
After running "make check", the outputs (to stdout and stderr) of test programs are stored
in test/*/seq_runs.sh.log. From there you will see those familiar outputs from 1.8.1.
In the new design, the outputs of all test programs in the same folder will go to the
same log file.
Running "make check" will stop immediately when encountering the first failed test and
print the failure messages on screen. An example of failed make check with error messages
is given below. If you did not see any error/failed message at the end, then all tests
were passed. To make that more obvious, I will add a print statement at the end of
"make" and "make check" to indicate whether libraries have been built and all test programs
have passed, respectively.
Please note the above changes only apply to "make check" which tests only serial programs,
i.e. running on one MPI process. For "make ptest" that tests parallel programs, the
reporting style is kept the same as 1.8.1. When encountering a failed test, "make ptest"
will report the failure on screen and continue to run the next test program.
Example of screen shot for failed "make check":
...
Making check in test
Making check in common
Making check in C
PASS: seq_runs.sh
============================================================================
Testsuite summary for parallel-netcdf 1.9.0.pre1
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
Making check in fandc
Making check in nc_test
FAIL: seq_runs.sh
============================================================================
Testsuite summary for parallel-netcdf 1.9.0.pre1
============================================================================
# TOTAL: 1
# PASS: 0
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See test/nc_test/test-suite.log
Please report to parallel-netcdf at mcs.anl.gov
============================================================================
make[5]: *** [test-suite.log] Error 1
make[4]: *** [check-TESTS] Error 2
make[3]: *** [check-am] Error 2
make[2]: *** [check] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check-recursive] Error 1
Wei-keng
On Nov 21, 2017, at 1:38 AM, Carl Ponder wrote:
> A question on the regression-tests: in this pre-release version, I see 10 regression-tests being reported
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 0
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> # TOTAL: 1
> whereas in 1.8.1, there were 86 tests reported:
> *** TESTING C nc_test for format CDF-1
> *** TESTING C nc_test for format CDF-2
> *** TESTING C nc_test for format CDF-5
> *** TESTING C t_nc for emulating netCDF t_nc ------ pass
> *** TESTING C tst_misc for emulating netCDF t_misc ------ pass
> *** TESTING C tst_norm for emulating netCDF tst_norm ------ pass
> *** TESTING C tst_small for emulating netCDF tst_small ------ pass
> *** TESTING C tst_names for emulating netCDF tst_names ------ pass
> *** TESTING C tst_atts3 for emulating netCDF tst_atts3 ------ pass
> *** TESTING C tst_atts for emulating netCDF tst_atts ------ pass
> *** TESTING C tst_nofill for fill/nofill modes ------ pass
> *** TESTING C pres_temp_4D_wr for writing file ------ pass
> *** TESTING C pres_temp_4D_rd for reading file ------ pass
> *** TESTING C mcoll_perf for mput/iput APIs ------ pass
> *** TESTING C test_bput for bput API ------ pass
> *** TESTING C interleaved for writing interleaved fileviews ------ pass
> *** TESTING C i_varn_int64 for iput/iget varn ------ pass
> *** TESTING C flexible_bput for flexible bput_varm ------ pass
> *** TESTING C wait_after_indep for ncmpi_end_indep_data ------ pass
> *** TESTING C req_all for NC_REQ_ALL ------ pass
> *** TESTING C i_varn_indef for iput/iget varn in define mode ------ pass
> *** TESTING C bput_varn for bput_varn ------ pass
> *** TESTING C column_wise for iput/iget interleaved access ------ pass
> *** TESTING F77 ./mcoll_testf77 for iput API ------ pass
> *** TESTING F77 ./test_bputf77 for bput_varm_real API ------ pass
> *** TESTING F90 ./mcoll_testf for nf90mpi_iput_var API ------ pass
> *** TESTING F90 ./test_bputf for bput_var ------ pass
> *** TESTING C test_inq_format for inquiring CDF file formats ------ pass
> *** TESTING C cdf_type for CDF-5 type in CDF-1 and 2 ------ pass
> *** TESTING C dim_cdf12 for defining dim in CDF-1/2 format ------ pass
> *** TESTING C ncmpi_vars_null_stride for NULL stride ------ pass
> *** TESTING C vectors for put_vara/get_vara ------ pass
> *** TESTING C collective_error for collective abort ------ pass
> *** TESTING C test_varm for get/put varm ------ pass
> *** TESTING C alignment_test for alignment ------ pass
> *** TESTING C flexible for flexible put and get ------ pass
> *** TESTING C flexible2 for flexible APIs ------ pass
> *** TESTING C flexible_varm for flexible varm APIs ------ pass
> *** TESTING C nonblocking for using ncmpi_iput_vara_int() ------ pass
> *** TESTING C noclobber for NC_NOCLOBBER and NC_EEXIST ------ pass
> *** TESTING C record for write records in reversed order ------ pass
> *** TESTING C inq_num_vars for no. record/fixed variables ------ pass
> *** TESTING C varn_int for ncmpi_put_varn_int_all() ------ pass
> *** TESTING C modes for file create/open modes
> *** TESTING C one_record for only one record variable ------ pass
> *** TESTING C inq_recsize for inquiring record size ------ pass
> *** TESTING C test_vard for flexible put and get ------ pass
> *** TESTING C varn_contig for put_varn with contig fileview ------ pass
> *** TESTING C ivarn for ncmpi_iput_varn_<type>() ------ pass
> *** TESTING C check_striping for strining info ------ pass
> *** TESTING C add_var for checking offsets of new variables ------ pass
> *** TESTING C buftype_free for free buftype in flexible API ------ pass
> *** TESTING C last_large_var for last large var in CDF-1/2 ------ pass
> *** TESTING C check_type for checking for type conflict ------ pass
> *** TESTING C test_erange for checking for NC_ERANGE ------ pass
> *** TESTING C scalar for get/put scalar variables ------ pass
> *** TESTING C redef1 for entering re-define mode ------ pass
> *** TESTING F77 ./varn_intf for varn API ------ pass
> *** TESTING F77 ./attrf for attribute overflow ------ pass
> *** TESTING F77 ./buftype_freef for flexible API ------ pass
> *** TESTING F77 ./put_parameter for using immutable write buf ------ pass
> *** TESTING F77 ./test_vardf for vard API ------ pass
> *** TESTING F90 ./inq_num_varsf for no. record/fixed variables ------ pass
> *** TESTING F90 ./inq_recsizef for inquiring record size ------ pass
> *** TESTING F90 ./test_vardf90 for vard API ------ pass
> *** TESTING F90 ./varn_real for varn API ------ pass
> *** TESTING C erange_fill for checking for type conflict ------ pass
> *** TESTING C redef1 for entering re-define mode ------ pass
> *** TESTING C++ nctst for APIs with different netCDF formats ------ pass
> *** TESTING C++ test_classic for creation of classic format file ------ pass
> *** TESTING F77 ./nf_test for CDF-1 ------ pass
> *** TESTING F77 ./nf_test for CDF-2 ------ pass
> *** TESTING F77 ./nf_test for CDF-5 ------ pass
> *** TESTING F90 ./nf90_test for CDF-1 ------ pass
> *** TESTING F90 ./nf90_test for CDF-2 ------ pass
> *** TESTING F90 ./nf90_test for CDF-5 ------ pass
> *** TESTING F90 ./test_intent for INTENT modifier ------ pass
> *** TESTING F90 ./tst_f90 ------ pass
> *** TESTING F90 ./f90tst_vars for def_var API ------ pass
> *** TESTING F90 ./tst_types2 for 64-bit integer types ------ pass
> *** TESTING F90 ./tst_f90_cdf5 ------ pass
> *** TESTING F90 ./f90tst_vars2 for def_var API ------ pass
> *** TESTING F90 ./f90tst_vars3 for def_var API ------ pass
> *** TESTING F90 ./f90tst_vars4 for def_var API ------ pass
> *** TESTING F90 ./tst_flarge for large files ------ pass
> *** TESTING F90 ./tst_io ------ pass
> In the 1.9.0 list, are you aggregating the the results of more than one test in the 1-total's?
> For example, it looks like you're compiling several tests here but running a single driver, and reporting it as a single test:
> Making check in cdf_format
> make[2]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
> make test_inq_format cdf_type dim_cdf12 tst_open_cdf5 tst_corrupt
> make[3]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
> CC test_inq_format.o
> CCLD test_inq_format
> CC cdf_type.o
> CCLD cdf_type
> CC dim_cdf12.o
> CCLD dim_cdf12
> CC tst_open_cdf5.o
> CCLD tst_open_cdf5
> CC tst_corrupt.o
> CCLD tst_corrupt
> make[3]: Leaving directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
> make check-TESTS
> make[3]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
> make[4]: Entering directory `/cm/extra/apps/PNetCDF/1.9.0.pre1/PGI-17.10_OpenMPI-1.10.7_CUDA-9.0.176_Power/distro/test/cdf_format'
> PASS: seq_runs.sh
> ============================================================================
> Testsuite summary for parallel-netcdf 1.9.0.pre1
> ============================================================================
> # TOTAL: 1
> # PASS: 1
> # SKIP: 0
> # XFAIL: 0
> # FAIL: 0
> # XPASS: 0
> # ERROR: 0
> ============================================================================
> This would make it difficult to identify any single failures.
>
> This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
More information about the parallel-netcdf
mailing list