<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi, Jim
<div class=""><br class="">
</div>
<div class="">Is the ncmpi_enddef the first enddef call after the file creation,</div>
<div class="">or after a ncmpi_redef?</div>
<div class=""><br class="">
</div>
<div class="">In the former case, there is no MPI communication in PnetCDF, except</div>
<div class="">for an MPI_Barrier. In the latter case, if the file header size expands,</div>
<div class="">existing variables need to be moved to higher offsets, which require</div>
<div class="">PnetCDF to call MPI collective reads and writes and thus leads to MPI_Issend.</div>
<div class=""><br class="">
</div>
<div class="">Can you try to get a coredump so to trace the call stacks?</div>
<div class=""><br class="">
</div>
<div class="">You can also enable PnetCDF safe mode which will make additional MPI</div>
<div class="">communication calls for debugging purpose. Sometimes it helps narrow</div>
<div class="">down the problem cause. It can be enabled by setting environment</div>
<div class="">variable PNETCDF_SAFE_MODE to 1.</div>
<div class=""><br class="">
<div class="">Wei-keng </div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 21, 2022, at 5:03 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" class="">jedwards@ucar.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
I am using pnetcdf 1.12.3 and getting an error when compiled with intel/19.1.1 and impi/19.0.9 on the TACC Frontera system</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
I am getting very little information to guide me in debugging the error. <br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
[785] Abort(634628) on node 785 (rank 785 in comm 0): Fatal error in PMPI_Issend: Invalid tag, error stack:<br class="">
[785] PMPI_Issend(156): MPI_Issend(buf=0x2b5c81edf40f, count=1025120, MPI_BYTE, dest=0, tag=1048814, comm=0xc40000d7, request=0x7f2002783540) failed<br class="">
[785] PMPI_Issend(95).: Invalid tag, value is 1048814<br class="">
TACC: MPI job exited with code: 4 <br class="">
TACC: Shutdown complete. Exiting. <br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br class="">
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
I can tell that I am in a call to ncmpi_enddef but not getting anything beyond that - any ideas? <br class="">
</div>
<br class="">
-- <br class="">
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div class="">
<div class="">Jim Edwards<br class="">
<br class="">
</div>
<font size="1" class="">CESM Software Engineer<br class="">
</font></div>
<font size="1" class="">National Center for Atmospheric Research<br class="">
</font></div>
<font size="1" class="">Boulder, CO</font> <br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>