<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBanner8xwn6v8 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButton8xwn6v8:hover, .pfptPrimaryButton8xwn6v8:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButton8xwn6v8:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Jim, Thank you for your reply. I use openmpi/5. 0. 3 under intel/2022. 1. 2. I have tried with pnetcdf versions 12 , 13 and 14 but get the same error. I really think that there is some inconsistency during writing the file and as I said in the</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBanner8xwn6v8" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBanner8xwn6v8" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBanner8xwn6v8" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBanner8xwn6v8" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr">Hi Jim,<div><br></div><div>Thank you for your reply. I use openmpi/5.0.3 under intel/2022.1.2. I have tried with pnetcdf versions 12 , 13 and 14 but get the same error. I really think that there is some inconsistency during writing the file and as I said in the previous email, this only happens on one specific cluster. Here's a result of a test script: Using pnetcdf and mpi I write a global variable named <b>var</b> that is an array consisting of the indices of each process, i.e., each process writes only its own index. Example if I use 96 processes, var is [1, 2, 3, 4, ..., 95, 96]. However when I try writing the same variable using , say, 384 processes, some of the values in the middle remain zero (default element value) instead of the respective process index. Like you said, I am pretty sure this is <b>NOT</b> an issue with pnetcdf but some other inconsistency within the cluster, but I just don't know how to identify it and thought maybe people on the mailing list might have encountered this before. Meanwhile I got my code to work by doing simple binary IO instead of pnetcdf files, which tells me the MPI in itself is also okay I guess?</div><div><br></div><div>I appreciate your time with this.</div><div><br></div><div>Best</div><div><br></div><div>Divyanshu</div><div><br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Fri, Jul 11, 2025 at 4:23 PM Jim Edwards <<a href="mailto:jedwards@ucar.edu">jedwards@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">Hi Divyanshu,</div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">From your description it sounds like the file written is corrupted? And when you say "The error also doesn't appear when I use fewer processes on a single node on the cluster." Do you</div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">mean use fewer processes to write the file or to read the file? It really sounds to me like an application problem and not a pnetcdf or mpi issue. I think that you may need to provide</div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">an exact description of how the problem is created including the system you are using, the mpi library and version as well as the pnetcdf version and the application. </div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif;color:rgb(56,118,29)">Jim</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 11, 2025 at 4:13 PM Divyanshu Gola <<a href="mailto:divyans@umich.edu" target="_blank">divyans@umich.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden">
Hi, This is a shot in the dark but I thought why not. The exact same code that I had been using until a few weeks ago gives me an error when I am trying to read restart files using PNetcdf. The error is Index exceeds dimension bounds, or Start+Count</div>
<div style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender
</div>
<div id="m_-1817963850905388508m_-5776720769699647019pfptBannerw6xolfu" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-weight:normal;font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization.
</div>
</div>
<div style="clear:both;display:block;line-height:0;font-size:0.01px;height:0px"> </div>
</div>
<u></u>
<div style="display:none;font-size:1px;color:rgb(255,255,255);line-height:1px;max-height:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerEnd</div>
<div dir="ltr">Hi,<div><br></div><div>This is a shot in the dark but I thought why not. </div><div><br></div><div>The exact same code that I had been using until a few weeks ago gives me an error when I am trying to read restart files using PNetcdf. The error is <b>Index exceeds dimension bounds, or Start+Count exceeds dimension bounds. </b>Based on days of debugging, I have narrowed it down to some problem during the writing of the restart files (and not the reading itself). All of these errors seem to originate from the way PNetcdf is built and the MPI file system used on the cluster (because I can run the same code on a different cluster), but I can't seem to identify the root cause. The error also doesn't appear when I use fewer processes on a single node on the cluster.</div><div><br></div><div>I know this is most likely not a bug in the PNetcdf library but something else, but I was wondering if people on this mailing list have encountered a similar issue.</div><div><br></div><div>Apologies for the long email and thanks</div><div><br></div><div>Divyanshu </div><div><br></div><div>Postdoctoral Researcher</div><div>University of Michigan</div></div>
</div></blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br></div><div>STAND UP FOR SCIENCE</div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><div><font size="1">Boulder, CO</font> </div><br></div></div>
</blockquote></div>