<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Hi Wei-keng,</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">I am also adding 2GiB as a limit on the local size of aggregated arrays in pio. </div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Thanks,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 4, 2017 at 4:26 PM, Wei-keng Liao <span dir="ltr"><<a href="mailto:wkliao@eecs.northwestern.edu" target="_blank">wkliao@eecs.northwestern.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, Jim<br>
<br>
I will create a new error code, named NC_EMAX_REQ, in PnetCDF.<br>
So, if the request size of an individual call to a get/put API or<br>
wait API is > 2GiB, then NC_EMAX_REQ will be thrown. Note the<br>
new size limit is per MPI process, not across all MPI processes.<br>
I guess this behavior will affect PIO. Feedbacks are welcomed.<br>
<span class="HOEnZb"><font color="#888888"><br>
Wei-keng<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Dec 4, 2017, at 10:10 AM, Jim Edwards wrote:<br>
<br>
> Yes I can confirm that the aggregated size exceeds 2GiB.<br>
><br>
> On Sun, Dec 3, 2017 at 3:35 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> Hi, Jim<br>
><br>
> I think the error you encountered is most likely due to the<br>
> aggregated size of nonblocking requests in the ncmpi_wait_all<br>
> call being larger then 2 GiB. Can you confirm this is your case?<br>
> ROMIO does not appear to work well for such cases. I am<br>
> thinking to make PnetCDF to bail out if this condition is<br>
> detected.<br>
><br>
> Wei-keng<br>
><br>
> On Dec 3, 2017, at 7:55 AM, Jim Edwards wrote:<br>
><br>
> > I see you already put up the PR to ROMIO - thanks.<br>
> ><br>
> > On Fri, Dec 1, 2017 at 10:52 PM, Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu">wkliao@eecs.northwestern.edu</a>> wrote:<br>
> > Hi, Jim<br>
> ><br>
> > After taking another look at your assertion error from ad_gpfs_aggrs.c,<br>
> > I believe you were hit by a ROMIO bug. I wrote a short test program that<br>
> > can cause a similar integer overflow error in ROMIO. The program's URL:<br>
> > <a href="https://trac.mcs.anl.gov/projects/parallel-netcdf/browser/trunk/test/largefile/large_coalesce.c" rel="noreferrer" target="_blank">https://trac.mcs.anl.gov/<wbr>projects/parallel-netcdf/<wbr>browser/trunk/test/largefile/<wbr>large_coalesce.c</a><br>
> ><br>
> > Look like the bug has been predicted based on the following comments at line 463<br>
> > in file ad_gpfs_aggrs.c:<br>
> > /* Possibly reconsider if buf_idx's are ok as int's, or should they be aints/offsets?<br>
> > They are used as memory buffer indices so it seems like the 2G limit is in effect */<br>
> ><br>
> > After I rebuilt MPICH by changing the data type of buf_idx from int to MPI_Aint,<br>
> > my test program ran fine. Would you like to create an github issue at MPICH repo?<br>
> ><br>
> ><br>
> > Wei-keng<br>
> ><br>
> > On Dec 1, 2017, at 8:07 PM, Wei-keng Liao wrote:<br>
> ><br>
> > > Hi, Jim,<br>
> > ><br>
> > > Yes, that is a bug. I have developed a fix. Please check out the<br>
> > > latest commit from PnetCDF SVN repo and let me know if it works for you.<br>
> > > Thanks for reporting.<br>
> > ><br>
> > > Wei-keng<br>
> > ><br>
> > > On Dec 1, 2017, at 4:43 PM, Jim Edwards wrote:<br>
> > ><br>
> > >> I think that I've found a bug in the prerelease in file ncmpio_wait.c<br>
> > >><br>
> > >> In coalescing blocklengths at line 2095<br>
> > >> <br>
> > >> if (ai - a_last_contig == blocklengths[last_contig_req])<br>
> > >> /* user buffer of request j is contiguous from j-1<br>
> > >> * we coalesce j to j-1 */<br>
> > >> blocklengths[last_contig_req] += blocklengths[j];<br>
> > >><br>
> > >> It's possible that blocklengths[last_contig_req] + blocklengths[j]; overflows the integer datatype.<br>
> > >> I tried to fix that by avoiding the coalescing:<br>
> > >><br>
> > >> if ((ai - a_last_contig == blocklengths[last_contig_req]) &&<br>
> > >> (blocklengths[last_contig_req] + blocklengths[j] > 0))<br>
> > >> /* user buffer of request j is contiguous from j-1<br>
> > >> * we coalesce j to j-1 */<br>
> > >> blocklengths[last_contig_req] += blocklengths[j];<br>
> > >><br>
> > >> but that leads to another overflow problem :<br>
> > >> ad_gpfs_aggrs.c:572: ADIOI_GPFS_Calc_my_req: Assertion `curr_idx == (int) curr_idx' failed.<br>
> > >><br>
> > >><br>
> > >><br>
> > >> --<br>
> > >> Jim Edwards<br>
> > >><br>
> > >> CESM Software Engineer<br>
> > >> National Center for Atmospheric Research<br>
> > >> Boulder, CO<br>
> > ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Jim Edwards<br>
> ><br>
> > CESM Software Engineer<br>
> > National Center for Atmospheric Research<br>
> > Boulder, CO<br>
><br>
><br>
><br>
><br>
> --<br>
> Jim Edwards<br>
><br>
> CESM Software Engineer<br>
> National Center for Atmospheric Research<br>
> Boulder, CO<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>
</div>