<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">I spent a little time trying to do this but gave up and went back to using cray profiling tools to get more info.</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">One thing really stands out to me:</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">This is for the fast write:<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: | number of write gaps = 2<br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: | ave write gap size = 9722924978<br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: <ins>--------------------------------------------------------</ins><br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: RESULT: write SUBSET 1 16 64 4060.0217755460 4.5714040530</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">And this is for the slow one:<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: | number of write gaps = 1020<br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: | ave write gap size = 19079761<br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: <ins>--------------------------------------------------------</ins><br>
<a href="http://dec1793.hsn.de.hpc.ucar.edu">dec1793.hsn.de.hpc.ucar.edu</a> 0: RESULT: write SUBSET 1 16 64 76.2558020443 243.3913158400</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">Do you understand? <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 8, 2023 at 11:50 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
I have revised the example program to add writes to scalar and record variables.
<div>Let me know if that works for you. URL again is below.<br>
<div><br>
</div>
<div><a href="https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c" target="_blank">https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c</a><br>
<div><br>
</div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 7, 2023, at 6:10 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
That example doesn't include record variables. Do you have a similar one with record vars?</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 7, 2023 at 4:32 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Hi, Jim
<div><br>
</div>
<div>To eliminate the overheads of PIO, I suggest to use this PnetCDF example program</div>
<div>and add a scalar variable to see if the same happens.</div>
<div><br>
</div>
<div><a href="https://urldefense.com/v3/__https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c__;!!Dq0X2DkFhyF93HkjWTBQKhk!RGlLkVUbuYrrGrSkShv42nz4KqtPJK0FiNzPuYKV-esdwU5UcgKr0xLvQpOooAfY4n2UMB8meSG2ZanhcYgGU_Q$" target="_blank">https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c</a><br>
</div>
<div></div>
<div><br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 7, 2023, at 4:28 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-Keng,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
The cb_nodes doesn't seem to be affected. <br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Not using independent mode doesn't seem to have helped. I have the pioperf program now writing two files. One with only
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
decomposed fields and one with one additional field, rundate, which is a string with the date in it.</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
The performance is drastically different:</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
IO tasks vars Mb/s Time (s)<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 256 64 12067.7548254854 25.1577347560 (without scalar)<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 256 64 286.4615089145 1059.8190875640 (with scalar)
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 7, 2023 at 1:47 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is that the reason for why cb_nodes is 1?<br>
Strange, because cb_nodes is set at the file open time.<br>
<br>
Entering the independent data mode in PnetCDF can be completely avoided<br>
if using the nonblocking APIs.<br>
<br>
I would suggest your codes to use the nonblocking APIs in the following way.<br>
<br>
/* for non-partitioned variables */<br>
if (rank == 0) {<br>
ncmpi_iput_var_int(fh, varid[0], data[0], &req[0]); /* write the whole variable */<br>
ncmpi_iput_var_int(fh, varid[1], data[1], &req[1]);<br>
...<br>
}<br>
/* for partitioned variables */<br>
ncmpi_iput_vara_int(fh, varid[j], data[j], starts[j], counts[j], &req[j]);<br>
...<br>
<br>
<br>
/* commit all posted nonblocking requests */<br>
ncmpi_wait_all(ncid, NC_REQ_ALL, NC_REQ_NULL, NULL);<br>
<br>
<br>
Wei-keng<br>
<br>
> On Aug 7, 2023, at 2:12 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
> <br>
> Hi Wei-Keng,<br>
> <br>
> I think that I've found the problem. In the model I am writing a number of scalar variables to the file as well as the decomposed variables.<br>
> for the scalar variables I use a code structure like:<br>
> <br>
> ncmpi_begin_indep_data(fh);<br>
> ncmpi_put_vars_int(fh, varid, start, count, stride, data);<br>
> ncmpi_end_indep_data(fh);<br>
> <br>
> In my pioperf test code I didn't write any scalars - this morning I added one and the write performance for the decomposed variables got very very
<br>
> bad. What can I do about it?<br>
> <br>
> Jim<br>
> <br>
> <br>
> -- <br>
> Jim Edwards<br>
> <br>
> CESM Software Engineer<br>
> National Center for Atmospheric Research<br>
> Boulder, CO <br>
<br>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>