<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi, Jim
<div><br>
</div>
<div>To eliminate the overheads of PIO, I suggest to use this PnetCDF example program</div>
<div>and add a scalar variable to see if the same happens.</div>
<div><br>
</div>
<div><a href="https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c">https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c</a><br>
</div>
<div></div>
<div><br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 7, 2023, at 4:28 PM, Jim Edwards <jedwards@ucar.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
Hi Wei-Keng,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
The cb_nodes doesn't seem to be affected. <br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
Not using independent mode doesn't seem to have helped. I have the pioperf program now writing two files. One with only
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
decomposed fields and one with one additional field, rundate, which is a string with the date in it.</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
The performance is drastically different:</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
IO tasks vars Mb/s Time (s)<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
RESULT: write SUBSET 1 256 64 12067.7548254854 25.1577347560 (without scalar)<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
RESULT: write SUBSET 1 256 64 286.4615089145 1059.8190875640 (with scalar)
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 7, 2023 at 1:47 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Is that the reason for why cb_nodes is 1?<br>
Strange, because cb_nodes is set at the file open time.<br>
<br>
Entering the independent data mode in PnetCDF can be completely avoided<br>
if using the nonblocking APIs.<br>
<br>
I would suggest your codes to use the nonblocking APIs in the following way.<br>
<br>
/* for non-partitioned variables */<br>
if (rank == 0) {<br>
ncmpi_iput_var_int(fh, varid[0], data[0], &req[0]); /* write the whole variable */<br>
ncmpi_iput_var_int(fh, varid[1], data[1], &req[1]);<br>
...<br>
}<br>
/* for partitioned variables */<br>
ncmpi_iput_vara_int(fh, varid[j], data[j], starts[j], counts[j], &req[j]);<br>
...<br>
<br>
<br>
/* commit all posted nonblocking requests */<br>
ncmpi_wait_all(ncid, NC_REQ_ALL, NC_REQ_NULL, NULL);<br>
<br>
<br>
Wei-keng<br>
<br>
> On Aug 7, 2023, at 2:12 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:<br>
> <br>
> Hi Wei-Keng,<br>
> <br>
> I think that I've found the problem. In the model I am writing a number of scalar variables to the file as well as the decomposed variables.<br>
> for the scalar variables I use a code structure like:<br>
> <br>
> ncmpi_begin_indep_data(fh);<br>
> ncmpi_put_vars_int(fh, varid, start, count, stride, data);<br>
> ncmpi_end_indep_data(fh);<br>
> <br>
> In my pioperf test code I didn't write any scalars - this morning I added one and the write performance for the decomposed variables got very very
<br>
> bad. What can I do about it?<br>
> <br>
> Jim<br>
> <br>
> <br>
> -- <br>
> Jim Edwards<br>
> <br>
> CESM Software Engineer<br>
> National Center for Atmospheric Research<br>
> Boulder, CO <br>
<br>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>