<div dir="ltr">Wei-keng,<div><br></div><div>Sorry I didn't get back to you on this. I will give your code a test with my current MPI stack. If it doesn't help find anything, I will try building the latest nightly build of mpich. After that, I will start digging into the MPI-IO code and try and track down the issue.</div><div><br></div><div>Thanks,</div><div>Craig</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 22, 2015 at 6:54 PM, Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@eecs.northwestern.edu" target="_blank">wkliao@eecs.northwestern.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, Craig<br>
<br>
I have to admit I ran out of ideas.<br>
Let me explain my suspicion about the possible fault on inconsistent striping hints.<br>
<br>
One of the error message shown here:<br>
<span class=""> Warning (inconsistent metadata): variable lrefine's begin (root=1048576, local=3072)<br>
<br>
</span>says that rank 0 calculates the variable lrefine's starting file offset 1048576<br>
and another process calculates 3072. If Lustre's file striping unit is known, then<br>
PnetCDF will use a number aligned with striping unit for a variable's starting offset.<br>
If file striping unit is not available, then PnetCDF will align it with 512 bytes.<br>
So, in your case, the file stripe unit is 1048576, meaning rank 0 did have a correct<br>
value from MPI-IO hint, but the other process did not.<br>
<br>
PnetCDF calls MPI_Info_get() to get striping_unit value and assumes all processes<br>
get the same value returned from the same MPI call.<br>
<br>
Do you have MPICH installed on the same machine? If this is also happening to MPICH,<br>
then it is mvapich. If not, then PnetCDF is at fault.<br>
<br>
I wonder if you would like to try another test program that is in PnetCDF (attached).<br>
<span class="HOEnZb"><font color="#888888"><br>
Wei-keng<br>
<br>
</font></span><br><br>
<br>
On Sep 22, 2015, at 5:30 PM, Craig Tierney - NOAA Affiliate wrote:<br>
<br>
> Wei-keng,<br>
><br>
> I wasn't able to trigger a problem. Here is the script I ran around your test case:<br>
><br>
> #!/bin/bash --login<br>
><br>
> module load newdefaults<br>
> module load intel<br>
> export PATH=/home/admin/software/apps/mvapich2/2.1-intel/bin/:${PATH}<br>
><br>
> PDIR=/lfs2/jetmgmt/Craig.Tierney/test<br>
><br>
> if [ ! -d $PDIR/ ]; then<br>
> mkdir $PDIR<br>
> fi<br>
><br>
> for s in 1 4; do<br>
> if [ ! -d $PDIR/d$s ]; then<br>
> mkdir $PDIR/d$s<br>
> fi<br>
> lfs setstripe -c $s $PDIR/d$s<br>
> lfs getstripe $PDIR/d$s<br>
><br>
> rm -f $PDIR/d$s/bigfile<br>
> dd if=/dev/zero of=$PDIR/d$s/bigfile bs=1024k count=1<br>
> lfs getstripe $PDIR/d$s/bigfile<br>
><br>
> echo "Checking d$s"<br>
> mpiexec.hydra -np 2 ./check_mpi_striping $PDIR/d$s/bigfile<br>
> done<br>
><br>
> Here are the results:<br>
><br>
> $ ./doit<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d1<br>
> stripe_count: 1 stripe_size: 1048576 stripe_offset: -1<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d1/bigfile<br>
> lmm_stripe_count: 1<br>
> lmm_stripe_size: 1048576<br>
> lmm_pattern: 1<br>
> lmm_layout_gen: 0<br>
> lmm_stripe_offset: 4<br>
> obdidx objid objid group<br>
> 4 15487258 0xec511a 0<br>
><br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1048576 bytes (1.0 MB) copied, 0.00228375 s, 459 MB/s<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d1/bigfile<br>
> lmm_stripe_count: 1<br>
> lmm_stripe_size: 1048576<br>
> lmm_pattern: 1<br>
> lmm_layout_gen: 0<br>
> lmm_stripe_offset: 7<br>
> obdidx objid objid group<br>
> 7 15421630 0xeb50be 0<br>
><br>
> Checking d1<br>
> Success: striping_unit=1048576 striping_factor=1<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d4<br>
> stripe_count: 4 stripe_size: 1048576 stripe_offset: -1<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d4/bigfile<br>
> lmm_stripe_count: 4<br>
> lmm_stripe_size: 1048576<br>
> lmm_pattern: 1<br>
> lmm_layout_gen: 0<br>
> lmm_stripe_offset: 17<br>
> obdidx objid objid group<br>
> 17 15361627 0xea665b 0<br>
> 42 15439375 0xeb960f 0<br>
> 0 15384104 0xeabe28 0<br>
> 2 15522060 0xecd90c 0<br>
><br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1048576 bytes (1.0 MB) copied, 0.00210304 s, 499 MB/s<br>
> /lfs2/jetmgmt/Craig.Tierney/test/d4/bigfile<br>
> lmm_stripe_count: 4<br>
> lmm_stripe_size: 1048576<br>
> lmm_pattern: 1<br>
> lmm_layout_gen: 0<br>
> lmm_stripe_offset: 12<br>
> obdidx objid objid group<br>
> 12 15345301 0xea2695 0<br>
> 37 15646009 0xeebd39 0<br>
> 41 15695216 0xef7d70 0<br>
> 18 15500412 0xec847c 0<br>
><br>
> Checking d4<br>
> Success: striping_unit=1048576 striping_factor=4<br>
><br>
><br>
> Craig<br>
><br>
><br>
><br>
> On Tue, Sep 22, 2015 at 3:10 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> Hi, Craig<br>
><br>
> From these outputs, I think it is most likely due to MPI-IO fails<br>
> to return the same file striping unit and factor values among all<br>
> MPI processes. I guess only root process gets the correct values.<br>
> Attached is a short MPI program to test this theory.<br>
> Could you test it using at least 2 processes on Lustre?<br>
><br>
> To compile:<br>
> mpicc -o check_mpi_striping check_mpi_striping.c<br>
> To run:<br>
> mpiexec -n 2 check_mpi_striping<br>
><br>
><br>
> Wei-keng<br>
><br>
><br>
><br>
><br>
> On Sep 22, 2015, at 2:34 PM, Craig Tierney - NOAA Affiliate wrote:<br>
><br>
> > Wei-keng,<br>
> ><br>
> > Here is the output from my run with PNETCDF_SAFE_MODE=1 on Lustre:<br>
> ><br>
> > [root@Jet:fe8 FLASH-IO]# mpiexec.hydra -env PNETCDF_SAFE_MODE 1 -np 4 ./flash_benchmark_io /lfs2/jetmgmt/Craig.Tierney/d1//flash_io_test_<br>
> > Warning (inconsistent metadata): variable lrefine's begin (root=1048576, local=3072)<br>
> > Warning (inconsistent metadata): variable nodetype's begin (root=2097152, local=4608)<br>
> > Warning (inconsistent metadata): variable gid's begin (root=3145728, local=6144)<br>
> > Warning (inconsistent metadata): variable coordinates's begin (root=4194304, local=25600)<br>
> > Warning (inconsistent metadata): variable blocksize's begin (root=5242880, local=33792)<br>
> > Warning (inconsistent metadata): variable bndbox's begin (root=6291456, local=41984)<br>
> > Warning (inconsistent metadata): variable dens's begin (root=7340032, local=57856)<br>
> > Warning (inconsistent metadata): variable velx's begin (root=18874368, local=10641920)<br>
> > Warning (inconsistent metadata): variable lrefine's begin (root=1048576, local=3072)<br>
> > Warning (inconsistent metadata): variable nodetype's begin (root=2097152, local=4608)<br>
> > Warning (inconsistent metadata): variable gid's begin (root=3145728, local=6144)<br>
> > Warning (inconsistent metadata): variable coordinates's begin (root=4194304, local=25600)<br>
> > Warning (inconsistent metadata): variable blocksize's begin (root=5242880, local=33792)<br>
> > Warning (inconsistent metadata): variable bndbox's begin (root=6291456, local=41984)<br>
> > Warning (inconsistent metadata): variable dens's begin (root=7340032, local=57856)<br>
> > Warning (inconsistent metadata): variable velx's begin (root=18874368, local=10641920)<br>
> > Warning (inconsistent metadata): variable vely's begin (root=30408704, local=21225984)<br>
> > Warning (inconsistent metadata): variable velz's begin (root=41943040, local=31810048)<br>
> > Warning (inconsistent metadata): variable pres's begin (root=53477376, local=42394112)<br>
> > Warning (inconsistent metadata): variable ener's begin (root=65011712, local=52978176)<br>
> > Warning (inconsistent metadata): variable temp's begin (root=76546048, local=63562240)<br>
> > Warning (inconsistent metadata): variable gamc's begin (root=88080384, local=74146304)<br>
> > Warning (inconsistent metadata): variable game's begin (root=99614720, local=84730368)<br>
> > Warning (inconsistent metadata): variable enuc's begin (root=111149056, local=95314432)<br>
> > Warning (inconsistent metadata): variable gpot's begin (root=122683392, local=105898496)<br>
> > Warning (inconsistent metadata): variable f1__'s begin (root=134217728, local=116482560)<br>
> > Warning (inconsistent metadata): variable f2__'s begin (root=145752064, local=127066624)<br>
> > Warning (inconsistent metadata): variable f3__'s begin (root=157286400, local=137650688)<br>
> > Warning (inconsistent metadata): variable lrefine's begin (root=1048576, local=3072)<br>
> > Warning (inconsistent metadata): variable nodetype's begin (root=2097152, local=4608)<br>
> > Warning (inconsistent metadata): variable gid's begin (root=3145728, local=6144)<br>
> > Warning (inconsistent metadata): variable coordinates's begin (root=4194304, local=25600)<br>
> > Warning (inconsistent metadata): variable blocksize's begin (root=5242880, local=33792)<br>
> > Warning (inconsistent metadata): variable bndbox's begin (root=6291456, local=41984)<br>
> > Warning (inconsistent metadata): variable dens's begin (root=7340032, local=57856)<br>
> > Warning (inconsistent metadata): variable velx's begin (root=18874368, local=10641920)<br>
> > Warning (inconsistent metadata): variable vely's begin (root=30408704, local=21225984)<br>
> > Warning (inconsistent metadata): variable velz's begin (root=41943040, local=31810048)<br>
> > Warning (inconsistent metadata): variable pres's begin (root=53477376, local=42394112)<br>
> > Warning (inconsistent metadata): variable ener's begin (root=65011712, local=52978176)<br>
> > Warning (inconsistent metadata): variable temp's begin (root=76546048, local=63562240)<br>
> > Warning (inconsistent metadata): variable gamc's begin (root=88080384, local=74146304)<br>
> > Warning (inconsistent metadata): variable game's begin (root=99614720, local=84730368)<br>
> > Warning (inconsistent metadata): variable enuc's begin (root=111149056, local=95314432)<br>
> > Warning (inconsistent metadata): variable gpot's begin (root=122683392, local=105898496)<br>
> > Warning (inconsistent metadata): variable f1__'s begin (root=134217728, local=116482560)<br>
> > Warning (inconsistent metadata): variable f2__'s begin (root=145752064, local=127066624)<br>
> > Warning (inconsistent metadata): variable f3__'s begin (root=157286400, local=137650688)<br>
> > Warning (inconsistent metadata): variable f4__'s begin (root=168820736, local=148234752)<br>
> > Warning (inconsistent metadata): variable f5__'s begin (root=180355072, local=158818816)<br>
> > Warning (inconsistent metadata): variable f6__'s begin (root=191889408, local=169402880)<br>
> > Warning (inconsistent metadata): variable vely's begin (root=30408704, local=21225984)<br>
> > Warning (inconsistent metadata): variable velz's begin (root=41943040, local=31810048)<br>
> > Warning (inconsistent metadata): variable pres's begin (root=53477376, local=42394112)<br>
> > Warning (inconsistent metadata): variable ener's begin (root=65011712, local=52978176)<br>
> > Warning (inconsistent metadata): variable temp's begin (root=76546048, local=63562240)<br>
> > Warning (inconsistent metadata): variable gamc's begin (root=88080384, local=74146304)<br>
> > Warning (inconsistent metadata): variable game's begin (root=99614720, local=84730368)<br>
> > Warning (inconsistent metadata): variable enuc's begin (root=111149056, local=95314432)<br>
> > Warning (inconsistent metadata): variable gpot's begin (root=122683392, local=105898496)<br>
> > Warning (inconsistent metadata): variable f1__'s begin (root=134217728, local=116482560)<br>
> > Warning (inconsistent metadata): variable f2__'s begin (root=145752064, local=127066624)<br>
> > Warning (inconsistent metadata): variable f3__'s begin (root=157286400, local=137650688)<br>
> > Warning (inconsistent metadata): variable f4__'s begin (root=168820736, local=148234752)<br>
> > Warning (inconsistent metadata): variable f5__'s begin (root=180355072, local=158818816)<br>
> > Warning (inconsistent metadata): variable f6__'s begin (root=191889408, local=169402880)<br>
> > Warning (inconsistent metadata): variable f7__'s begin (root=203423744, local=179986944)<br>
> > Warning (inconsistent metadata): variable f8__'s begin (root=214958080, local=190571008)<br>
> > Warning (inconsistent metadata): variable f9__'s begin (root=226492416, local=201155072)<br>
> > Warning (inconsistent metadata): variable f10_'s begin (root=238026752, local=211739136)<br>
> > Warning (inconsistent metadata): variable f11_'s begin (root=249561088, local=222323200)<br>
> > Warning (inconsistent metadata): variable f12_'s begin (root=261095424, local=232907264)<br>
> > Warning (inconsistent metadata): variable f13_'s begin (root=272629760, local=243491328)<br>
> > Warning (inconsistent metadata): variable f4__'s begin (root=168820736, local=148234752)<br>
> > Warning (inconsistent metadata): variable f5__'s begin (root=180355072, local=158818816)<br>
> > Warning (inconsistent metadata): variable f6__'s begin (root=191889408, local=169402880)<br>
> > Warning (inconsistent metadata): variable f7__'s begin (root=203423744, local=179986944)<br>
> > Warning (inconsistent metadata): variable f8__'s begin (root=214958080, local=190571008)<br>
> > Warning (inconsistent metadata): variable f9__'s begin (root=226492416, local=201155072)<br>
> > Warning (inconsistent metadata): variable f10_'s begin (root=238026752, local=211739136)<br>
> > Warning (inconsistent metadata): variable f11_'s begin (root=249561088, local=222323200)<br>
> > Warning (inconsistent metadata): variable f12_'s begin (root=261095424, local=232907264)<br>
> > Warning (inconsistent metadata): variable f13_'s begin (root=272629760, local=243491328)<br>
> > Warning (inconsistent metadata): variable f7__'s begin (root=203423744, local=179986944)<br>
> > Warning (inconsistent metadata): variable f8__'s begin (root=214958080, local=190571008)<br>
> > Warning (inconsistent metadata): variable f9__'s begin (root=226492416, local=201155072)<br>
> > Warning (inconsistent metadata): variable f10_'s begin (root=238026752, local=211739136)<br>
> > Warning (inconsistent metadata): variable f11_'s begin (root=249561088, local=222323200)<br>
> > Warning (inconsistent metadata): variable f12_'s begin (root=261095424, local=232907264)<br>
> > Warning (inconsistent metadata): variable f13_'s begin (root=272629760, local=243491328)<br>
> > Here: -250<br>
> > Here: -262<br>
> > Here: -262<br>
> > Here: -262<br>
> > nfmpi_enddefFile header is inconsistent among processes<br>
> > nfmpi_enddef<br>
> > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > r<br>
> > nfmpi_enddef<br>
> > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > r<br>
> > nfmpi_enddef<br>
> > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > r<br>
> > [cli_1]: aborting job:<br>
> > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1<br>
> > [cli_0]: [cli_2]: aborting job:<br>
> > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2<br>
> > [cli_3]: aborting job:<br>
> > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3<br>
> > aborting job:<br>
> > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0<br>
> ><br>
> > Craig<br>
> ><br>
> > On Mon, Sep 21, 2015 at 1:21 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> ><br>
> > It is strange that the test failed for Lustre.<br>
> ><br>
> > The error message says some variables defined across MPI processes are not consistent.<br>
> > Could you run this benchmark with safe mode on? by setting the environment variable<br>
> > PNETCDF_SAFE_MODE to 1 before the run. This will print more error messages, such as<br>
> > which variables are inconsistent and at what offsets.<br>
> ><br>
> ><br>
> > Wei-keng<br>
> ><br>
> > On Sep 21, 2015, at 1:31 PM, Craig Tierney - NOAA Affiliate wrote:<br>
> ><br>
> > > Rob and Wei-keng,<br>
> > ><br>
> > > Thanks for you help on this problem. Rob - The patch seems to work. I had to hand apply it but now the pnetcdf tests (mostly) complete successfully. The FLASH-IO benchmark is failing when Lustre is used. It completes successfully when Panasas is used. The error code that is returned by nfmpi_enddef is -262. The description for this error is:<br>
> > ><br>
> > > #define NC_EMULTIDEFINE_VAR_BEGIN (-262) /**< inconsistent variable file begin offset (internal use) */<br>
> > ><br>
> > > [root@Jet:fe7 FLASH-IO]# mpiexec.hydra -n 4 ./flash_benchmark_io /pan2/jetmgmt/Craig.Tierney/pan_flash_io_test_<br>
> > > Here: 0<br>
> > > Here: 0<br>
> > > Here: 0<br>
> > > Here: 0<br>
> > > number of guards : 4<br>
> > > number of blocks : 80<br>
> > > number of variables : 24<br>
> > > checkpoint time : 12.74 sec<br>
> > > max header : 0.88 sec<br>
> > > max unknown : 11.83 sec<br>
> > > max close : 0.53 sec<br>
> > > I/O amount : 242.30 MiB<br>
> > > plot no corner : 2.38 sec<br>
> > > max header : 0.59 sec<br>
> > > max unknown : 1.78 sec<br>
> > > max close : 0.22 sec<br>
> > > I/O amount : 20.22 MiB<br>
> > > plot corner : 2.52 sec<br>
> > > max header : 0.81 sec<br>
> > > max unknown : 1.51 sec<br>
> > > max close : 0.96 sec<br>
> > > I/O amount : 24.25 MiB<br>
> > > -------------------------------------------------------<br>
> > > File base name : /pan2/jetmgmt/Craig.Tierney/pan_flash_io_test_<br>
> > > file striping count : 0<br>
> > > file striping size : 301346992 bytes<br>
> > > Total I/O amount : 286.78 MiB<br>
> > > -------------------------------------------------------<br>
> > > nproc array size exec (sec) bandwidth (MiB/s)<br>
> > > 4 16 x 16 x 16 17.64 16.26<br>
> > ><br>
> > ><br>
> > > [root@Jet:fe7 FLASH-IO]# mpiexec.hydra -n 4 ./flash_benchmark_io /lfs2/jetmgmt/Craig.Tierney/lfs_flash_io_test_<br>
> > > Here: -262<br>
> > > Here: -262<br>
> > > Here: -262<br>
> > > nfmpi_enddef<br>
> > > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > > r<br>
> > > nfmpi_enddef<br>
> > > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > > r<br>
> > > nfmpi_enddef<br>
> > > (Internal error) beginning file offset of this variable is inconsistent among p<br>
> > > r<br>
> > > Here: 0<br>
> > > [cli_1]: aborting job:<br>
> > > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1<br>
> > > [cli_3]: [cli_2]: aborting job:<br>
> > > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3<br>
> > > aborting job:<br>
> > > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2<br>
> > ><br>
> > > ===================================================================================<br>
> > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES<br>
> > > = PID 16702 RUNNING AT fe7<br>
> > > = EXIT CODE: 255<br>
> > > = CLEANING UP REMAINING PROCESSES<br>
> > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES<br>
> > > ===================================================================================<br>
> > ><br>
> > > Thanks,<br>
> > > Craig<br>
> > ><br>
> > ><br>
> > > On Mon, Sep 21, 2015 at 8:30 AM, Rob Latham <<a href="mailto:robl@mcs.anl.gov">robl@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > ><br>
> > > On 09/20/2015 03:44 PM, Craig Tierney - NOAA Affiliate wrote:<br>
> > > Wei-keng,<br>
> > ><br>
> > > I tried your test code on a different system, and I found it worked with<br>
> > > Intel+mvapich2 (2.1rc1). That system was using Panasas and I was<br>
> > > testing on Lustre. I then tried Panasas on the original machine<br>
> > > (supports both Panasas and Lustre) and I got the correct behavior.<br>
> > ><br>
> > > So the problem somehow related to Lustre. We are using the 2.5.37.ddn<br>
> > > client. Unless you have an obvious answer, I will open this with DDN<br>
> > > tomorrow.<br>
> > ><br>
> > ><br>
> > > Ah, bet I know why this is!<br>
> > ><br>
> > > the Lustre driver and (some versions of the) Panasas driver set their fs-specific hints by opening the file, setting some ioctls, then continuing on without deleting the file.<br>
> > ><br>
> > > In the common case, when we expect the file to show up, no one notices or cares, but in MPI_MODE_EXCL or some other restrictive flags, the file gets created when we did not expect it to -- and that's part of the reason this bug lived on so long.<br>
> > ><br>
> > > I fixed this by moving file manipulations out of the hint parsing path and into the open path (after we check permissions and flags)<br>
> > ><br>
> > > Relevant commit: <a href="https://trac.mpich.org/projects/mpich/changeset/92f1c69f0de87f9" rel="noreferrer" target="_blank">https://trac.mpich.org/projects/mpich/changeset/92f1c69f0de87f9</a><br>
> > ><br>
> > > See more details from Darshan, OpenMPI, and MPICH here:<br>
> > > - <a href="https://trac.mpich.org/projects/mpich/ticket/2261" rel="noreferrer" target="_blank">https://trac.mpich.org/projects/mpich/ticket/2261</a><br>
> > > - <a href="https://github.com/open-mpi/ompi/issues/158" rel="noreferrer" target="_blank">https://github.com/open-mpi/ompi/issues/158</a><br>
> > > - <a href="http://lists.mcs.anl.gov/pipermail/darshan-users/2015-February/000256.html" rel="noreferrer" target="_blank">http://lists.mcs.anl.gov/pipermail/darshan-users/2015-February/000256.html</a><br>
> > ><br>
> > > ==rob<br>
> > ><br>
> > ><br>
> > > Thanks,<br>
> > > Craig<br>
> > ><br>
> > > On Sun, Sep 20, 2015 at 2:36 PM, Craig Tierney - NOAA Affiliate<br>
> > > <<a href="mailto:craig.tierney@noaa.gov">craig.tierney@noaa.gov</a> <mailto:<a href="mailto:craig.tierney@noaa.gov">craig.tierney@noaa.gov</a>>> wrote:<br>
> > ><br>
> > > Wei-keng,<br>
> > ><br>
> > > Thanks for the test case. Here is what I get using a set of<br>
> > > compilers and MPI stacks. I was expecting that mvapich2 1.8 and 2.1<br>
> > > would behave differently.<br>
> > ><br>
> > > What versions of MPI do you test internally?<br>
> > ><br>
> > > Craig<br>
> > ><br>
> > > Testing intel+impi<br>
> > ><br>
> > > Currently Loaded Modules:<br>
> > > 1) newdefaults 2) intel/<a href="http://15.0.3.187" rel="noreferrer" target="_blank">15.0.3.187</a> <<a href="http://15.0.3.187" rel="noreferrer" target="_blank">http://15.0.3.187</a>> 3)<br>
> > > impi/<a href="http://5.1.1.109" rel="noreferrer" target="_blank">5.1.1.109</a> <<a href="http://5.1.1.109" rel="noreferrer" target="_blank">http://5.1.1.109</a>><br>
> > ><br>
> > > Error at line 22: File does not exist, error stack:<br>
> > > ADIOI_NFS_OPEN(69): File /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>> does not exist<br>
> > > Testing intel+mvapich2 2.1<br>
> > ><br>
> > > Currently Loaded Modules:<br>
> > > 1) newdefaults 2) intel/<a href="http://15.0.3.187" rel="noreferrer" target="_blank">15.0.3.187</a> <<a href="http://15.0.3.187" rel="noreferrer" target="_blank">http://15.0.3.187</a>> 3)<br>
> > > mvapich2/2.1<br>
> > ><br>
> > > file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>><br>
> > > Testing intel+mvapich2 1.8<br>
> > ><br>
> > > Currently Loaded Modules:<br>
> > > 1) newdefaults 2) intel/<a href="http://15.0.3.187" rel="noreferrer" target="_blank">15.0.3.187</a> <<a href="http://15.0.3.187" rel="noreferrer" target="_blank">http://15.0.3.187</a>> 3)<br>
> > > mvapich2/1.8<br>
> > ><br>
> > > file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>><br>
> > > Testing pgi+mvapich2 2.1<br>
> > ><br>
> > > Currently Loaded Modules:<br>
> > > 1) newdefaults 2) pgi/15.3 3) mvapich2/2.1<br>
> > ><br>
> > > file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>><br>
> > > Testing pgi+mvapich2 1.8<br>
> > ><br>
> > > Currently Loaded Modules:<br>
> > > 1) newdefaults 2) pgi/15.3 3) mvapich2/1.8<br>
> > ><br>
> > > file was opened: /lfs3/jetmgmt/Craig.Tierney/<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>><br>
> > ><br>
> > > Craig<br>
> > ><br>
> > > On Sun, Sep 20, 2015 at 1:43 PM, Wei-keng Liao<br>
> > > <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a> <mailto:<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>>><br>
> > > wrote:<br>
> > ><br>
> > > In that case, it is likely mvapich does not perform correctly.<br>
> > ><br>
> > > In PnetCDF, when NC_NOWRITE is used in a call to ncmpi_open,<br>
> > > PnetCDF calls a MPI_File_open with the open flag set to<br>
> > > MPI_MODE_RDONLY. See<br>
> > > <a href="http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/tags/v1-6-1/src/lib/mpincio.c#L322" rel="noreferrer" target="_blank">http://trac.mcs.anl.gov/projects/parallel-netcdf/browser/tags/v1-6-1/src/lib/mpincio.c#L322</a><br>
> > ><br>
> > > Maybe test this with a simple MPI-IO program below.<br>
> > > It prints error messages like<br>
> > > Error at line 15: File does not exist, error stack:<br>
> > > ADIOI_UFS_OPEN(69): File <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>> does not exist<br>
> > ><br>
> > > But, no file should be created.<br>
> > ><br>
> > ><br>
> > > #include <stdio.h><br>
> > > #include <unistd.h> /* unlink() */<br>
> > > #include <mpi.h><br>
> > ><br>
> > > int main(int argc, char **argv) {<br>
> > > int err;<br>
> > > MPI_File fh;<br>
> > ><br>
> > > MPI_Init(&argc, &argv);<br>
> > ><br>
> > > /* delete "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>>" and<br>
> > > ignore the error */<br>
> > > unlink("<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>>");<br>
> > ><br>
> > > err = MPI_File_open(MPI_COMM_WORLD, "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>>", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);<br>
> > > if (err != MPI_SUCCESS) {<br>
> > > int errorStringLen;<br>
> > > char errorString[MPI_MAX_ERROR_STRING];<br>
> > > MPI_Error_string(err, errorString, &errorStringLen);<br>
> > > printf("Error at line %d: %s\n",__LINE__, errorString);<br>
> > > }<br>
> > > else<br>
> > > MPI_File_close(&fh);<br>
> > ><br>
> > > MPI_Finalize();<br>
> > > return 0;<br>
> > > }<br>
> > ><br>
> > ><br>
> > > Wei-keng<br>
> > ><br>
> > > On Sep 20, 2015, at 1:51 PM, Craig Tierney - NOAA Affiliate wrote:<br>
> > ><br>
> > > > Wei-keng,<br>
> > > ><br>
> > > > I always run distclean before I try to build the code. The<br>
> > > first test failing is nc_test. The problem seems to be in this<br>
> > > test:<br>
> > > ><br>
> > > > err = ncmpi_open(comm, "<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a><br>
> > > <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>>", NC_NOWRITE, info, &ncid);/* should fail */<br>
> > > > IF (err == NC_NOERR)<br>
> > > > error("ncmpi_open of nonexistent file should have<br>
> > > failed");<br>
> > > > IF (err != NC_ENOENT)<br>
> > > > error("ncmpi_open of nonexistent file should have<br>
> > > returned NC_ENOENT");<br>
> > > > else {<br>
> > > > /* printf("Expected error message complaining: \"File<br>
> > > <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>> does not exist\"\n"); */<br>
> > > > nok++;<br>
> > > > }<br>
> > > ><br>
> > > > A zero length <a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">tooth-fairy.nc</a> <<a href="http://tooth-fairy.nc" rel="noreferrer" target="_blank">http://tooth-fairy.nc</a>> file is<br>
> > > being created, and I don't think that is supposed to happen.<br>
> > > That would mean that the mode NC_NOWRITE is not being honored by<br>
> > > MPI_IO. I will look at this more tomorrow and try to craft a<br>
> > > short example.<br>
> > > ><br>
> > > > Craig<br>
> > > ><br>
> > > > On Sun, Sep 20, 2015 at 10:23 AM, Wei-keng Liao<br>
> > > <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a><br>
> > > <mailto:<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>>> wrote:<br>
> > > > Hi, Craig<br>
> > > ><br>
> > > > Your config.log looks fine to me.<br>
> > > > Some of your error messages are supposed to report errors of<br>
> > > opening<br>
> > > > a non-existing file, but report a different error code,<br>
> > > meaning the<br>
> > > > file does exist. I suspect it may be because of residue files.<br>
> > > ><br>
> > > > Could you do a clean rebuild with the following commands?<br>
> > > > % make -s distclean<br>
> > > > % ./configure --prefix=/apps/pnetcdf/1.6.1-intel-mvapich2<br>
> > > > % make -s -j8<br>
> > > > % make -s check<br>
> > > ><br>
> > > > If the problem persists, then it might be because mvapich.<br>
> > > ><br>
> > > > Wei-keng<br>
> > > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Rob Latham<br>
> > > Mathematics and Computer Science Division<br>
> > > Argonne National Lab, IL USA<br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
><br>
<br>
<br></blockquote></div><br></div>