<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 16px; font-family: Arial, sans-serif; ">
<div>
<div><br>
</div>
<div>Hi Wei-keng:</div>
<div><br>
</div>
<div>Your program (successfully?) triggers two errors when I run it with 'mpiexec_mpt –np 2 ./mpi_io_hint'</div>
<div><br>
</div>
<div>(1) This error message:</div>
<div><br>
</div>
<div>
<div>MPI_File_f2c: Invalid file handle</div>
<div> Error at MPI_File_close Invalid file handle</div>
</div>
<div><br>
</div>
<div>(2) The program then hangs and must be manually killed.</div>
<div><br>
</div>
<div>The same behavior occurs when increasing the number of MPI processes. However, when I only run with 1 process (serially), I only see the error message (1) but the program stops gracefully. These errors are with SGI MPT 2.12.</div>
<div><br>
</div>
<div>The program runs w/o issue if I use Intel MPI 15.0.3.187. </div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>-Eric</div>
<div><br>
</div>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Lucida Grande; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, February 3, 2016 2:32 PM<br>
<span style="font-weight:bold">To: </span>Eric Kemp <<a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: Problems with testing PNETCDF 1.6.1<br>
</div>
<div><br>
</div>
<div>
<div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hi, Eric<br>
<br>
Many thanks for digging the problem. It is great to know the cause.<br>
<br>
In PnetCDF, writing the file header is done by root process using an<br>
independent write. So, this line is not necessary. However, setting<br>
this hint should not cause the program to hang or break anything.<br>
<br>
This indicates a bug in the MPI-IO component of the SGI MPT.<br>
<br>
Attached is a small test program that should reproduce the hang<br>
problem you encountered.<br>
<br>
Thanks again for put more effort on this!<br>
<br>
Wei-keng<br>
<br>
</div>
</span></font></div>
<div class="BodyFragment"><font size="2"><span style="font-size:10pt;">
<div class="PlainText"><br>
On Feb 3, 2016, at 12:41 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:<br>
<br>
> <br>
> Hi Wei-keng:<br>
> <br>
> Good news: I went through the source code, adding print statements, and<br>
> traced the hang to the nfmpi_wait_all invocation in<br>
> checkpoint_ncmpi_parallel.F90. After some further trial-and-error, I<br>
> fixed the problem by commenting out this line in subroutine<br>
> checkpoint_wr_ncmpi_par:<br>
> <br>
> call MPI_Info_set(file_info, 'romio_no_indep_rw', 'true', err)<br>
> <br>
> I don't understand what is going on under the hood, but perhaps the<br>
> combination of SGI MPT and GPFS (the file system on NASA's supercomputer)<br>
> makes this setting undesirable?<br>
> <br>
> Anyway, after commented out this line, both SGI MPT and Intel MPI tests<br>
> work, and produce FLASH output files with the appropriate sizes. So I'm<br>
> going to declare victory and install with this change.<br>
> <br>
> Thanks for your help!<br>
> <br>
> -Eric<br>
> <br>
> Eric M. Kemp (SSAI)<br>
> NASA/GSFC <br>
> Mail Code: 606 <br>
> Greenbelt, MD 20771<br>
> 301.286.9768 <br>
> <a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a><br>
> <a href="mailto:eric.kemp@ssaihq.com">eric.kemp@ssaihq.com</a><br>
> <br>
> <br>
> <br>
> On 2/3/16 11:24 AM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> <br>
>> Hi, Eric<br>
>> <br>
>> The file size 2964 is the file header size supposed to be. This<br>
>> indicates the program has reached the call to nfmpi_enddef() at<br>
>> line 155 of file checkpoint_ncmpi_parallel.F90.<br>
>> The file size also tells that no data has been written to the file yet.<br>
>> In this case, two possible places can cause the program to hang:<br>
>> line 155 and line 600.<br>
>> <br>
>> If you put a print statement after line 156, then it can check if<br>
>> the program returns from it correctly or hangs at nfmpi_enddef().<br>
>> <br>
>> The other possible hangs is at line 600, the call to nfmpi_wait_all.<br>
>> <br>
>> Wei-keng<br>
>> <br>
>> On Feb 3, 2016, at 7:08 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br>
>> AND APPLICATIONS INC] wrote:<br>
>> <br>
>>> <br>
>>> Good morning Wei-keng:<br>
>>> <br>
>>> I reran 1.7.0.pre1 FLASH-IO again. The only output from the program is:<br>
>>> <br>
>>> rw-r--r-- 1 emkemp k3002 2964 Feb 3 07:53<br>
>>> flash_io_test_ncmpi_chk_0000.nc<br>
>>> <br>
>>> I will tinker with the source code to see if I can identify where it<br>
>>> hangs.<br>
>>> <br>
>>> <br>
>>> Thanks,<br>
>>> <br>
>>> -Eric<br>
>>> <br>
>>> <br>
>>> <br>
>>> On 2/2/16 5:36 PM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
>>> <br>
>>>> Hi, Eric<br>
>>>> <br>
>>>> Unfortunately, I do not have access to an SGI machine. We usually<br>
>>>> rely on our users to do some initial debugging for the situation like<br>
>>>> this.<br>
>>>> I know this can be too much to ask, but if you did not encounter any<br>
>>>> problem when running your program, maybe you can ignore this test.<br>
>>>> <br>
>>>> However, since the hanging occurs only with SGI MPT, I suspect it is<br>
>>>> related to MPT.<br>
>>>> <br>
>>>> Could you check one thing for me? After you kill the FLASH-IO job,<br>
>>>> could<br>
>>>> you check if any netCDF files were created? The expected files and<br>
>>>> their<br>
>>>> sizes are<br>
>>>> <br>
>>>> -rw------- 1 254075392 Feb 1 21:58 flash_io_test_ncmpi_chk_0000.nc<br>
>>>> -rw------- 1 21208576 Feb 1 21:58 flash_io_test_ncmpi_plt_cnt_0000.nc<br>
>>>> -rw------- 1 25431372 Feb 1 21:58 flash_io_test_ncmpi_plt_crn_0000.nc<br>
>>>> <br>
>>>> <br>
>>>> Wei-keng<br>
>>>> <br>
>>>> On Feb 2, 2016, at 3:53 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br>
>>>> AND APPLICATIONS INC] wrote:<br>
>>>> <br>
>>>>> <br>
>>>>> Hi Wei-keng:<br>
>>>>> <br>
>>>>> I tried rerunning the entire installation with PNETCDF_SAFE_MODE=1.<br>
>>>>> FLASH-IO still hangs with SGI MPT (with no error message), but it<br>
>>>>> works<br>
>>>>> fine with Intel MPI.<br>
>>>>> <br>
>>>>> -Eric<br>
>>>>> <br>
>>>>> <br>
>>>>> From: Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>><br>
>>>>> Date: Tuesday, February 2, 2016 12:39 PM<br>
>>>>> To: Eric Kemp <<a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a>><br>
>>>>> Cc: "<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>><br>
>>>>> Subject: Re: Problems with testing PNETCDF 1.6.1<br>
>>>>> <br>
>>>>> Hi, Eric<br>
>>>>> <br>
>>>>> Sorry for sending the wrong file. The correct one is attached, in case<br>
>>>>> you would like<br>
>>>>> to use it.<br>
>>>>> <br>
>>>>> I check your config.log file but could not find any thing fishy.<br>
>>>>> I just now tested it with Intel compiler 16.0.0.109 without a problem.<br>
>>>>> Could you try running FLASH-IO under the safe mode? i.e. set the<br>
>>>>> environment<br>
>>>>> variable PNETCDF_SAFE_MODE to 1. It will enable internal checking for<br>
>>>>> data inconsistency.<br>
>>>>> <br>
>>>>> Just want to make sure for 1.7.0.pre1 that your "make ptest" failed<br>
>>>>> only on FLASH-IO.<br>
>>>>> Because FLAH-IO is the last test program, this means all other tests<br>
>>>>> have passed.<br>
>>>>> Let me know. Thanks.<br>
>>>>> <br>
>>>>> Wei-keng<br>
>>>>> <br>
>>>>> <br>
>>>>> On Feb 2, 2016, at 8:27 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br>
>>>>> AND APPLICATIONS INC] wrote:<br>
>>>>> <br>
>>>>>> <br>
>>>>>> Hi Wei-keng:<br>
>>>>>> <br>
>>>>>> I think you sent me the wrong copy of that file ‹ it was identical to<br>
>>>>> what is in 1.7.0.pre1. But I went ahead and added "cmd" as an<br>
>>>>> argument<br>
>>>>> to subroutine check_err, and that test code compiles and runs.<br>
>>>>>> <br>
>>>>>> The large file tests pass in 1.7.0.pre1 as you indicated. However,<br>
>>>>> FLASH-IO still hangs with SGI MPT. I took your suggestion and tried<br>
>>>>> running this test separately (cd benchmarks/FLASH-IO ; make ptest) but<br>
>>>>> the code still hangs.<br>
>>>>>> <br>
>>>>>> I've attached the (gzipped) config.log file from the 1.7.0pre1<br>
>>>>> installations.<br>
>>>>>> <br>
>>>>>> Thanks,<br>
>>>>>> <br>
>>>>>> -Eric<br>
>>>>>> <br>
>>>>>> Eric M. Kemp (SSAI)<br>
>>>>>> NASA/GSFC <br>
>>>>>> Mail Code: 606<br>
>>>>>> Greenbelt, MD 20771<br>
>>>>>> 301.286.9768 <br>
>>>>>> <a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a><br>
>>>>>> <a href="mailto:eric.kemp@ssaihq.com">eric.kemp@ssaihq.com</a><br>
>>>>>> <br>
>>>>>> <br>
>>>>>> From: Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>><br>
>>>>>> Date: Monday, February 1, 2016 5:12 PM<br>
>>>>>> To: Eric Kemp <<a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a>><br>
>>>>>> Cc: "<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov">parallel-netcdf@mcs.anl.gov</a>><br>
>>>>>> Subject: Re: Problems with testing PNETCDF 1.6.1<br>
>>>>>> <br>
>>>>>> Hi, Eric,<br>
>>>>>> <br>
>>>>>> Thanks for reporting the error. This is another oversight, Sorry.<br>
>>>>>> The fixed file, bigrecords.f, is attached.<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> Wei-keng<br>
>>>>>> <br>
>>>>>> <br>
>>>>>> <br>
>>>>>> On Feb 1, 2016, at 2:41 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE<br>
>>>>> SYSTEMS AND APPLICATIONS INC] wrote:<br>
>>>>>> <br>
>>>>>>> <br>
>>>>>>> Hi Wei-keng:<br>
>>>>>>> <br>
>>>>>>> Thanks for your quick response. I tried installing 1.7.0.pre1 but I<br>
>>>>> get a<br>
>>>>>>> different error when compiling the tests:<br>
>>>>>>> <br>
>>>>>>> /usr/local/intel/2016/impi/5.1.2.150/bin64/mpif90 -I../../src/lib<br>
>>>>>>> -I./../common -I../../src/libf -I../../src/libf90 -fpic -O2<br>
>>>>> -fp-model<br>
>>>>>>> strict -c bigrecords.f<br>
>>>>>>> bigrecords.f(333): error #6514: A substring must be of type<br>
>>>>> CHARACTER.<br>
>>>>>>> [CMD]<br>
>>>>>>> msg = '*** TESTING F77 '//cmd(1:XTRIM(cmd))//<br>
>>>>>>> ------------------------------------^<br>
>>>>>>> bigrecords.f(333): error #6054: A CHARACTER data type is required<br>
>>>>> in this<br>
>>>>>>> context. [CMD]<br>
>>>>>>> msg = '*** TESTING F77 '//cmd(1:XTRIM(cmd))//<br>
>>>>>>> ------------------------------------^<br>
>>>>>>> compilation aborted for bigrecords.f (code 1)<br>
>>>>>>> <br>
>>>>>>> <br>
>>>>>>> <br>
>>>>>>> This appears to be a legitimate syntax error in the test program, in<br>
>>>>>>> subroutine check_err. "cmd" is not defined in that subroutine, nor<br>
>>>>> is it<br>
>>>>>>> a global variable.<br>
>>>>>>> <br>
>>>>>>> I will try patching 1.6.1 with the NC_64BIT_DATA constant instead.<br>
>>>>>>> <br>
>>>>>>> -Eric<br>
>>>>>>> <br>
>>>>>>> <br>
>>>>>>> <br>
>>>>>>> <br>
>>>>>>> On 2/1/16 12:02 PM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>><br>
>>>>> wrote:<br>
>>>>>>> <br>
>>>>>>>> Hi, Eric<br>
>>>>>>>> <br>
>>>>>>>> For the large file tests, the error is caused by a oversight of<br>
>>>>> using a<br>
>>>>>>>> wrong flag.<br>
>>>>>>>> Line 81 of file large_files.c should have used NC_64BIT_DATA,<br>
>>>>> instead of<br>
>>>>>>>> NC_64BIT_OFFSET.<br>
>>>>>>>> This error has been fixed in the pre-release of 1.7.0.pre1. Could<br>
>>>>> you<br>
>>>>>>>> give it a try?<br>
>>>>>>>> <a href="http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html">
http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html</a><br>
>>>>>>>> <br>
>>>>>>>> As for the FLASH-IO test, could you try running it alone? I.e. cd<br>
>>>>> to the<br>
>>>>>>>> folder<br>
>>>>>>>> benchmarks/FLASH-IO and run "make ptest" there. In the meantime,<br>
>>>>> please<br>
>>>>>>>> send me<br>
>>>>>>>> the file config.log.<br>
>>>>>>>> <br>
>>>>>>>> <br>
>>>>>>>> Wei-keng<br>
>>>>>>>> <br>
>>>>>>>> On Feb 1, 2016, at 7:32 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE<br>
>>>>> SYSTEMS<br>
>>>>>>>> AND APPLICATIONS INC] wrote:<br>
>>>>>>>> <br>
>>>>>>>>> <br>
>>>>>>>>> Dear PNETCDF developers:<br>
>>>>>>>>> <br>
>>>>>>>>> I'm attempting to install PNETCDF1.6.1 on a Linux cluster running<br>
>>>>> SLES<br>
>>>>>>>>> 11.3. I'm using Intel 15 Fortran and C compilers (no C++), and<br>
>>>>> I'm<br>
>>>>>>>>> trying to install for two separate MPI implementations (SGI MPT<br>
>>>>> 2.12 and<br>
>>>>>>>>> Intel MPI 5.1.2).<br>
>>>>>>>>> <br>
>>>>>>>>> I'm encountering two problems when I run 'make ptest'.<br>
>>>>>>>>> <br>
>>>>>>>>> 1) For both MPI implementations, the large file tests fail with<br>
>>>>> an<br>
>>>>>>>>> integer overflow. The error message is:<br>
>>>>>>>>> <br>
>>>>>>>>> *** Testing large files, slowly.<br>
>>>>>>>>> line 116 of large_files.c: Overflow when type cast to 4-byte<br>
>>>>> integer.<br>
>>>>>>>>> *** Creating large file ./testfile.nc...srun.slurm: error:<br>
>>>>> borgo018:<br>
>>>>>>>>> task 0: Exited with exit code 1<br>
>>>>>>>>> <br>
>>>>>>>>> I reviewed the README.large_files for guidance, and I can confirm<br>
>>>>> that<br>
>>>>>>>>> both 'MPI_Offset' and 'off_t' are 8 bytes.<br>
>>>>>>>>> <br>
>>>>>>>>> 2) For SGI MPT only, if I disable support for large file tests,<br>
>>>>> 'make<br>
>>>>>>>>> ptest' hangs when testing FLASH-IO:<br>
>>>>>>>>> <br>
>>>>>>>>> make -w -C FLASH-IO ptest<br>
>>>>>>>>> make[2]: Entering directory<br>
>>>>>>>>> <br>
>>>>> <br>
>>>>> `/gpfsm/dnb32/emkemp/NUWRFLIB/svn/trunk/builds/parallel-netcdf-1.6.1/be<br>
>>>>> nc<br>
>>>>>>>>> hmarks/FLASH-IO'<br>
>>>>>>>>> mpiexec_mpt -n 4 ./flash_benchmark_io ./flash_io_test_<br>
>>>>>>>>> srun.slurm: cluster configuration lacks support for cpu binding<br>
>>>>>>>>> <br>
>>>>>>>>> The earlier tests with both single and multiple processes work<br>
>>>>> for SGI<br>
>>>>>>>>> MPT. And all tests (again, excluding large file tests) work for<br>
>>>>> Intel<br>
>>>>>>>>> MPI.<br>
>>>>>>>>> <br>
>>>>>>>>> I can provide more information (e.g., output from the configure<br>
>>>>> script)<br>
>>>>>>>>> upon request.<br>
>>>>>>>>> <br>
>>>>>>>>> Thanks,<br>
>>>>>>>>> <br>
>>>>>>>>> -Eric<br>
>>>>>>>>> <br>
>>>>>>>>> Eric M. Kemp (SSAI)<br>
>>>>>>>>> NASA/GSFC <br>
>>>>>>>>> Mail Code: 606<br>
>>>>>>>>> Greenbelt, MD 20771<br>
>>>>>>>>> 301.286.9768<br>
>>>>>>>>> <a href="mailto:eric.kemp@nasa.gov">eric.kemp@nasa.gov</a><br>
>>>>>>>>> <a href="mailto:eric.kemp@ssaihq.com">eric.kemp@ssaihq.com</a><br>
>>>>>>>>> <br>
>>>>>>>> <br>
>>>>>>> <br>
>>>>>> <br>
>>>>>> <config.log.gz><br>
>>>>> <br>
>>>> <br>
>>> <br>
>> <br>
> <br>
<br>
</div>
</span></font></div>
</div>
</div>
</span>
</body>
</html>