<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">John,</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">I think that the latest pnetcdf version is 1.6.1 - I know that there was at least one change specifically for openmpi</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">pnetcdf is a very easy build, I recommend trying a newer version.</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Jim</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 10, 2015 at 12:27 PM, John Michalakes <span dir="ltr"><<a href="mailto:john.michalakes@noaa.gov" target="_blank">john.michalakes@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="#0563C1" vlink="#954F72"><div><p class="MsoNormal">Hi,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Having a problem where an <span>MPI</span> Fortran program (<span>WRF</span>) can open a file for writing using <span>NFMPI_CREATE</span> when all tasks are on one node, but fails if the tasks are spread over multiple nodes.<span>  </span>Have isolated to a small test program:<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">Program hello<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>implicit none<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>include "<span>mpif.h</span>"<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">#include "<span>pnetcdf.inc</span>"<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>integer<span>                           </span>:: <span>stat,Status</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>integer<span>                           </span>:: info, <span>ierr</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>integer <span>Comm</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>integer <span>ncid</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>CALL <span>MPI_INIT</span>( <span>ierr</span> )<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span><span>Comm</span> = <span>MPI_COMM_WORLD</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>call <span>mpi_info_create</span>( info, <span>ierr</span> )<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>CALL <span>mpi_info_set</span>(<span>info,"romio_ds_write","disable</span>", <span>ierr</span>) ;<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">write(0,*)'<span>mpi_info_set</span> write returns ',<span>ierr</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>CALL <span>mpi_info_set</span>(<span>info,"romio_ds_read","disable</span>", <span>ierr</span>) ;<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">write(0,*)'<span>mpi_info_set</span> read returns ',<span>ierr</span><u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>stat = <span>NFMPI_CREATE</span>(<span>Comm</span>, '<span>testfile_d01</span>', <span>IOR</span>(<span>NF_CLOBBER</span>, <span>NF_64BIT_OFFSET</span>), info, <span>NCID</span>)<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">write(0,*)'after <span>NFMPI_CREATE</span> ', stat<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>call <span>mpi_info_free</span>( info, <span>ierr</span> )<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>stat = <span>NFMPI_CLOSE</span>(<span>NCID</span>)<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">write(0,*)'after <span>NFMPI_CLOSE</span> ', stat<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>CALL <span>MPI_FINALIZE</span>( <span>ierr</span> )<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""><span>  </span>STOP<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">End Program hello<u></u><u></u></span></p><p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New""> <u></u><u></u></span></p><p class="MsoNormal">Running with two tasks on a single node this generates:<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal" style="margin-left:.5in"><span>a515</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"><span>a515</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> write returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> read returns<span>        </span><span>    </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> write returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> read returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CREATE</span><span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CREATE</span><span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CLOSE</span><span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CLOSE</span><span>            </span>0<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">But running with 2 tasks, each on a separate node:<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal" style="margin-left:.5in"><span>a811</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"><span>a817</span><u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> write returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> read returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> write returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> <span>mpi_info_set</span> read returns<span>            </span>0<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CREATE</span><span>         </span>-208<span>   </span><<<<<<<<<<<<<<<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CLOSE</span><span>          </span>-33<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CREATE</span><span>         </span>-208<span>  </span><<<<<<<<<<<<<<<u></u><u></u></p><p class="MsoNormal" style="margin-left:.5in"> after <span>NFMPI_CLOSE</span><span>          </span>-33<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I have tested the program on other systems such as <span>NCAR’s</span> Yellowstone and it works fine on any combination of nodes.<span>  </span>This target system is a user’s system running <span>openmpi</span>/1.6.3 compiled for intel.<span>  </span>The version of <span>pnetcdf</span> is 1.3.1.<span>  </span>I’m pretty sure it’s a <span>Lustre</span> file system (but will have to follow up with the user and their support staff to be sure).<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I’m assuming there’s some misconfiguration or installation of <span>MPI</span> or <span>pNetCDF</span> on the user’s system, but I need some help with how to proceed.<span>  </span>Thanks,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">John <u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><i><span style="font-size:12.0pt">John Michalakes<br>Scientific Programmer/Analyst<u></u><u></u></span></i></p><p class="MsoNormal"><i><span style="font-size:12.0pt">National Centers for Environmental Prediction<u></u><u></u></span></i></p><p class="MsoNormal"><i><span style="font-size:12.0pt"><a href="mailto:john.michalakes@noaa.gov" target="_blank">john.michalakes@noaa.gov</a><u></u><u></u></span></i></p><p class="MsoNormal"><i><span style="font-size:12.0pt"><a href="tel:301-683-3847" value="+13016833847" target="_blank">301-683-3847</a><u></u><u></u></span></i></p><p class="MsoNormal"><u></u> <u></u></p></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>
</div>