<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi, Jim
<div class=""><br class="">
</div>
<div class="">Thanks for the update.</div>
<div class=""><br class="">
</div>
<div class="">I am wondering if your I/O pattern produces a large number</div>
<div class="">of noncontiguous file access requests in each MPI process.</div>
<div class="">Because ROMIO used MPI tags in its implementation of 2-phase I/O,</div>
<div class="">this pattern can result in a large number of MPI isend/irecv,</div>
<div class="">each uses a unique MPI tag. The latest ROMIO has fixed this for</div>
<div class="">Lustre (<a href="https://github.com/pmodels/mpich/pull/5660" class="">https://github.com/pmodels/mpich/pull/5660</a>.)</div>
<div class=""><br class="">
<div class="">Wei-keng </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 28, 2022, at 9:41 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" class="">jedwards@ucar.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
Hi Wei-Keng,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
I found the issue with help from TACC user support:<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<a href="https://urldefense.com/v3/__https://www.intel.com/content/www/us/en/developer/articles/technical/large-mpi-tags-with-the-intel-mpi.html__;!!Dq0X2DkFhyF93HkjWTBQKhk!VT-ST0BuSkBPBDMQPJpRTv8AHEeTsBLq5E4LV_kjmYjFUDJoc3Z0jCGiDj3ItqkQ5t8Pgm2wZa23ViXRQ8jKf48$" target="_blank" class="">https://www.intel.com/content/www/us/en/developer/articles/technical/large-mpi-tags-with-the-intel-mpi.html</a></div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">I
set Setting Environment MPIR_CVAR_CH4_OFI_RANK_BITS=15</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial" class="">
<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class=""> Setting
Environment MPIR_CVAR_CH4_OFI_TAG_BITS=24</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial" class="">
<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">and
added a print statement:</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial" class="">
<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">cam_restart.F90
123 Maximum tag value queried 8388607</span><br style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial" class="">
<span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none" class="">this
appears to be working. <br class="">
</span></div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Jun 21, 2022 at 7:25 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank" class="">wkliao@northwestern.edu</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="">Hi, Jim
<div class=""><br class="">
</div>
<div class="">Is the ncmpi_enddef the first enddef call after the file creation,</div>
<div class="">or after a ncmpi_redef?</div>
<div class=""><br class="">
</div>
<div class="">In the former case, there is no MPI communication in PnetCDF, except</div>
<div class="">for an MPI_Barrier. In the latter case, if the file header size expands,</div>
<div class="">existing variables need to be moved to higher offsets, which require</div>
<div class="">PnetCDF to call MPI collective reads and writes and thus leads to MPI_Issend.</div>
<div class=""><br class="">
</div>
<div class="">Can you try to get a coredump so to trace the call stacks?</div>
<div class=""><br class="">
</div>
<div class="">You can also enable PnetCDF safe mode which will make additional MPI</div>
<div class="">communication calls for debugging purpose. Sometimes it helps narrow</div>
<div class="">down the problem cause. It can be enabled by setting environment</div>
<div class="">variable PNETCDF_SAFE_MODE to 1.</div>
<div class=""><br class="">
<div class="">Wei-keng </div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 21, 2022, at 5:03 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank" class="">jedwards@ucar.edu</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I am using pnetcdf 1.12.3 and getting an error when compiled with intel/19.1.1 and impi/19.0.9 on the TACC Frontera system</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I am getting very little information to guide me in debugging the error. <br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
[785] Abort(634628) on node 785 (rank 785 in comm 0): Fatal error in PMPI_Issend: Invalid tag, error stack:<br class="">
[785] PMPI_Issend(156): MPI_Issend(buf=0x2b5c81edf40f, count=1025120, MPI_BYTE, dest=0, tag=1048814, comm=0xc40000d7, request=0x7f2002783540) failed<br class="">
[785] PMPI_Issend(95).: Invalid tag, value is 1048814<br class="">
TACC: MPI job exited with code: 4 <br class="">
TACC: Shutdown complete. Exiting. <br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
I can tell that I am in a call to ncmpi_enddef but not getting anything beyond that - any ideas? <br class="">
</div>
<br class="">
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">Jim Edwards<br class="">
<br class="">
</div>
<font size="1" class="">CESM Software Engineer<br class="">
</font></div>
<font size="1" class="">National Center for Atmospheric Research<br class="">
</font></div>
<font size="1" class="">Boulder, CO</font> <br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<br class="">
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">Jim Edwards<br class="">
<br class="">
</div>
<font size="1" class="">CESM Software Engineer<br class="">
</font></div>
<font size="1" class="">National Center for Atmospheric Research<br class="">
</font></div>
<font size="1" class="">Boulder, CO</font> <br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>