<div dir="ltr">Wei-keng,<div><br></div><div>I tried your test code on a different system, and I found it worked with Intel+mvapich2 (2.1rc1). That system was using Panasas and I was testing on Lustre. I then tried Panasas on the original machine (supports both Panasas and Lustre) and I got the correct behavior. </div><div><br></div><div>So the problem somehow related to Lustre. We are using the 2.5.37.ddn client. Unless you have an obvious answer, I will open this with DDN tomorrow.</div><div><br></div><div>Thanks,</div><div>Craig</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 20, 2015 at 2:36 PM, Craig Tierney - NOAA Affiliate <span dir="ltr"><<a href="mailto:craig.tierney@noaa.gov" target="_blank">craig.tierney@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Wei-keng,<div><br></div><div>Thanks for the test case. Here is what I get using a set of compilers and MPI stacks. I was expecting that mvapich2 1.8 and 2.1 would behave differently. </div><div><br></div><div>What versions of MPI do you test internally? </div><div><br></div><div>Craig</div><div><br></div><div><div>Testing intel+impi</div><div><br></div><div>Currently Loaded Modules:</div><div> 1) newdefaults 2) intel/<a href="http://15.0.3.187" target="_blank">15.0.3.187</a> 3) impi/<a href="http://5.1.1.109" target="_blank">5.1.1.109</a></div><div><br></div><div>Error at line 22: File does not exist, error stack:</div><div>ADIOI_NFS_OPEN(69): File /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" target="_blank">tooth-fairy.nc</a> does not exist</div><div>Testing intel+mvapich2 2.1</div><div><br></div><div>Currently Loaded Modules:</div><div> 1) newdefaults 2) intel/<a href="http://15.0.3.187" target="_blank">15.0.3.187</a> 3) mvapich2/2.1</div><div><br></div><div>file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" target="_blank">tooth-fairy.nc</a></div><div>Testing intel+mvapich2 1.8</div><div><br></div><div>Currently Loaded Modules:</div><div> 1) newdefaults 2) intel/<a href="http://15.0.3.187" target="_blank">15.0.3.187</a> 3) mvapich2/1.8</div><div><br></div><div>file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" target="_blank">tooth-fairy.nc</a></div><div>Testing pgi+mvapich2 2.1</div><div><br></div><div>Currently Loaded Modules:</div><div> 1) newdefaults 2) pgi/15.3 3) mvapich2/2.1</div><div><br></div><div>file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" target="_blank">tooth-fairy.nc</a></div><div>Testing pgi+mvapich2 1.8</div><div><br></div><div>Currently Loaded Modules:</div><div> 1) newdefaults 2) pgi/15.3 3) mvapich2/1.8</div><div><br></div><div>file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" target="_blank">tooth-fairy.nc</a></div></div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Craig</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 20, 2015 at 1:43 PM, Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@eecs.northwestern.edu" target="_blank">wkliao@eecs.northwestern.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">In that case, it is likely mvapich does not perform correctly.<br>
<br>
In PnetCDF, when NC_NOWRITE is used in a call to ncmpi_open,<br>
PnetCDF calls a MPI_File_open with the open flag set to MPI_MODE_RDONLY. See<br>
<a href="http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/tags/v1-6-1/src/lib/mpincio.c#L322" rel="noreferrer" target="_blank">http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/tags/v1-6-1/src/lib/mpincio.c#L322</a><br>
<br>
Maybe test this with a simple MPI-IO program below.<br>
It prints error messages like<br>
Error at line 15: File does not exist, error stack:<br>
ADIOI_UFS_OPEN(69): File <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> does not exist<br>
<br>
But, no file should be created.<br>
<br>
<br>
#include <stdio.h><br>
#include <unistd.h> /* unlink() */<br>
#include <mpi.h><br>
<br>
int main(int argc, char **argv) {<br>
int err;<br>
MPI_File fh;<br>
<br>
MPI_Init(&argc, &argv);<br>
<br>
/* delete "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a>" and ignore the error */<br>
unlink("<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a>");<br>
<br>
err = MPI_File_open(MPI_COMM_WORLD, "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a>", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);<br>
if (err != MPI_SUCCESS) {<br>
int errorStringLen;<br>
char errorString[MPI_MAX_ERROR_STRING];<br>
MPI_Error_string(err, errorString, &errorStringLen);<br>
printf("Error at line %d: %s\n",__LINE__, errorString);<br>
}<br>
else<br>
MPI_File_close(&fh);<br>
<br>
MPI_Finalize();<br>
return 0;<br>
}<br>
<span><font color="#888888"><br>
<br>
Wei-keng<br>
</font></span><div><div><br>
On Sep 20, 2015, at 1:51 PM, Craig Tierney - NOAA Affiliate wrote:<br>
<br>
> Wei-keng,<br>
><br>
> I always run distclean before I try to build the code. The first test failing is nc_test. The problem seems to be in this test:<br>
><br>
> err = ncmpi_open(comm, "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a>", NC_NOWRITE, info, &ncid);/* should fail */<br>
> IF (err == NC_NOERR)<br>
> error("ncmpi_open of nonexistent file should have failed");<br>
> IF (err != NC_ENOENT)<br>
> error("ncmpi_open of nonexistent file should have returned NC_ENOENT");<br>
> else {<br>
> /* printf("Expected error message complaining: \"File <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> does not exist\"\n"); */<br>
> nok++;<br>
> }<br>
><br>
> A zero length <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> file is being created, and I don't think that is supposed to happen. That would mean that the mode NC_NOWRITE is not being honored by MPI_IO. I will look at this more tomorrow and try to craft a short example.<br>
><br>
> Craig<br>
><br>
> On Sun, Sep 20, 2015 at 10:23 AM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" target="_blank">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> Hi, Craig<br>
><br>
> Your config.log looks fine to me.<br>
> Some of your error messages are supposed to report errors of opening<br>
> a non-existing file, but report a different error code, meaning the<br>
> file does exist. I suspect it may be because of residue files.<br>
><br>
> Could you do a clean rebuild with the following commands?<br>
> % make -s distclean<br>
> % ./configure --prefix=/apps/pnetcdf/1.6.1-intel-mvapich2<br>
> % make -s -j8<br>
> % make -s check<br>
><br>
> If the problem persists, then it might be because mvapich.<br>
><br>
> Wei-keng<br>
><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>