<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerkol84br { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonkol84br:hover, .pfptPrimaryButtonkol84br:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonkol84br:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hello, Thank you for your email. I tried running the test script using the debug flags and I get the following messages on all ranks, even though the code is able to execute after that: ---------------BEGIN ERROR MESSAGES-------------- Rank</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerkol84br" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerkol84br" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerkol84br" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerkol84br" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hello,<div><br></div><div>Thank you for your email. I tried running the test script using the debug flags and I get the following messages on all ranks, even though the code is able to execute after that:</div><div><br></div><div>---------------BEGIN ERROR MESSAGES--------------</div><div>Rank 144: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 146: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 147: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 148: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 149: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 150: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 151: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 152: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 154: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 156: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 157: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 158: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 116: NC_EBADDIM error at line 95 of NC_finddim in ncmpio_dim.c<br>Rank 152: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 152: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 81: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 81: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 89: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 89: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 102: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 102: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 103: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 103: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 112: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 112: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 114: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 114: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 119: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 119: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 122: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 122: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 124: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c<br>Rank 124: NC_ENOTVAR error at line 480 of ncmpio_inq_varid in ncmpio_var.c<br>Rank 136: NC_ENOTVAR error at line 259 of NC_findvar in ncmpio_var.c</div><div>-------------END ERROR MESSAGES---------------</div><div><br></div><div>Thanks for your help with this.</div><div><br></div><div>Divyanshu</div><div><br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Jul 14, 2025 at 12:38 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
Hi, Divyanshu
<div><br>
</div>
<div>When PnetCDF is configured with option "--enable-debug" and</div>
<div>the environment variable PNETCDF_VERBOSE_DEBUG_MODE is set to 1,</div>
<div>additional error messages describing the bound violation will</div>
<div>be printed on screen. It may help you find the source code location</div>
<div>that produces the error.</div>
<div><br id="m_-8889013899535635977lineBreakAtBeginningOfMessage">
<div>
<div>Wei-keng</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Jul 14, 2025, at 7:56 AM, Divyanshu Gola <<a href="mailto:divyans@umich.edu" target="_blank">divyans@umich.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr" id="m_-8889013899535635977pfptBanner8xwn6v8" style="writing-mode:revert;color:revert;font-family:revert;font-feature-settings:revert;font-kerning:revert;font-size:revert;font-size-adjust:revert;font-variant-alternates:revert;font-variant-caps:revert;font-variant-east-asian:revert;font-variant-ligatures:revert;font-variant-numeric:revert;font-weight:revert;text-orientation:revert;zoom:revert;letter-spacing:revert;background-blend-mode:revert;background-clip:revert;background-image:revert;background-origin:revert;background-position:revert;background-repeat:revert;background-size:revert;border-collapse:revert;box-sizing:revert;break-after:revert;break-before:revert;break-inside:revert;caption-side:revert;clear:revert;columns:revert;column-fill:revert;column-rule:revert;column-span:revert;empty-cells:revert;float:revert;image-orientation:revert;isolation:revert;line-break:revert;line-height:revert;list-style:revert;mix-blend-mode:revert;object-fit:revert;object-position:revert;outline:revert;overflow:revert;quotes:revert;table-layout:revert;text-align:left;text-align-last:revert;text-combine-upright:revert;text-decoration:revert;text-emphasis:revert;text-indent:revert;text-overflow:revert;text-transform:revert;text-underline-position:revert;vertical-align:revert;white-space:revert;word-break:revert;border-spacing:revert;word-spacing:revert;display:block;opacity:revert;background-color:rgb(208,216,220);border-bottom:revert;border-left:revert;border-right:revert;height:revert;max-height:revert;max-width:revert;min-height:revert;width:revert;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;border-top:4px solid rgb(144,164,174)">
<div id="m_-8889013899535635977pfptBanner8xwn6v8" style="writing-mode:unset;color:unset;font-family:unset;font-feature-settings:unset;font-kerning:unset;font-size:unset;font-size-adjust:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background:unset;background-blend-mode:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:left;image-orientation:unset;isolation:unset;line-break:unset;line-height:unset;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-combine-upright:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;display:block;opacity:unset;border-radius:unset;height:unset;max-height:unset;min-height:unset;min-width:unset;padding:unset;width:unset;margin:0px 0px 1px;max-width:600px">
<div id="m_-8889013899535635977pfptBanner8xwn6v8" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:14px;font-size-adjust:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:bold;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-clip:unset;background-image:unset;background-origin:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-combine-upright:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;display:block;opacity:unset;background-color:rgb(208,216,220);border-radius:unset;height:unset;margin:unset;max-height:unset;max-width:unset;min-height:unset;min-width:unset;padding:unset;width:unset">
This Message Is From an External Sender</div>
<div id="m_-8889013899535635977pfptBanner8xwn6v8" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:12px;font-size-adjust:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-clip:unset;background-image:unset;background-origin:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-combine-upright:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;display:block;opacity:unset;background-color:rgb(208,216,220);border-radius:unset;height:unset;margin-bottom:unset;margin-left:unset;margin-right:unset;max-height:unset;max-width:unset;min-height:unset;min-width:unset;padding:unset;width:unset;margin-top:2px">
This message came from outside your organization.</div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px">
</div>
</div>
<div dir="ltr" style="font-family:Monaco;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
Hi Jim,
<div><br>
</div>
<div>Thank you for your reply. I use openmpi/5.0.3 under intel/2022.1.2. I have tried with pnetcdf versions 12 , 13 and 14 but get the same error. I really think that there is some inconsistency during writing the file and as I said in the previous email, this
only happens on one specific cluster. Here's a result of a test script: Using pnetcdf and mpi I write a global variable named<span> </span><b>var</b> that is an array consisting of the indices of each process, i.e., each process
writes only its own index. Example if I use 96 processes, var is [1, 2, 3, 4, ..., 95, 96]. However when I try writing the same variable using , say, 384 processes, some of the values in the middle remain zero (default element value) instead of the respective
process index. Like you said, I am pretty sure this is<span> </span><b>NOT</b> an issue with pnetcdf but some other inconsistency within the cluster, but I just don't know how to identify it and thought maybe people on the mailing
list might have encountered this before. Meanwhile I got my code to work by doing simple binary IO instead of pnetcdf files, which tells me the MPI in itself is also okay I guess?</div>
<div><br>
</div>
<div>I appreciate your time with this.</div>
<div><br>
</div>
<div>Best</div>
<div><br>
</div>
<div>Divyanshu</div>
<div><br>
</div>
</div>
<br style="font-family:Monaco;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div class="gmail_quote" style="font-family:Monaco;font-size:14px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr" class="gmail_attr">On Fri, Jul 11, 2025 at 4:23 PM Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
Hi Divyanshu,</div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
>From your description it sounds like the file written is corrupted? And when you say "The error also doesn't appear when I use fewer processes on a single node on the cluster." Do you</div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
mean use fewer processes to write the file or to read the file? It really sounds to me like an application problem and not a pnetcdf or mpi issue. I think that you may need to provide</div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
an exact description of how the problem is created including the system you are using, the mpi library and version as well as the pnetcdf version and the application. </div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">
Jim</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jul 11, 2025 at 4:13 PM Divyanshu Gola <<a href="mailto:divyans@umich.edu" target="_blank">divyans@umich.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><u></u>
<div dir="ltr" id="m_-8889013899535635977m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-8889013899535635977m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-8889013899535635977m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;background-color:rgb(208,216,220);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender</div>
<div id="m_-8889013899535635977m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;background-color:rgb(208,216,220);font-weight:normal;font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization.</div>
</div>
<div style="clear:both;display:block;line-height:0;font-size:0.01px;height:0px">
</div>
</div>
<u></u>
<div dir="ltr">Hi,
<div><br>
</div>
<div>This is a shot in the dark but I thought why not. </div>
<div><br>
</div>
<div>The exact same code that I had been using until a few weeks ago gives me an error when I am trying to read restart files using PNetcdf. The error is<span> </span><b>Index exceeds dimension bounds, or Start+Count exceeds dimension
bounds.<span> </span></b>Based on days of debugging, I have narrowed it down to some problem during the writing of the restart files (and not the reading itself). All of these errors seem to originate from the way PNetcdf is built
and the MPI file system used on the cluster (because I can run the same code on a different cluster), but I can't seem to identify the root cause. The error also doesn't appear when I use fewer processes on a single node on the cluster.</div>
<div><br>
</div>
<div>I know this is most likely not a bug in the PNetcdf library but something else, but I was wondering if people on this mailing list have encountered a similar issue.</div>
<div><br>
</div>
<div>Apologies for the long email and thanks</div>
<div><br>
</div>
<div>Divyanshu </div>
<div><br>
</div>
<div>Postdoctoral Researcher</div>
<div>University of Michigan</div>
</div>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<br>
<span class="gmail_signature_prefix">--<span> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
</div>
<div>STAND UP FOR SCIENCE</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<div><font size="1">Boulder, CO</font></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div>