performance issue

[Wei-Keng Liao]

Hi, Jim

To eliminate the overheads of PIO, I suggest to use this PnetCDF example program
and add a scalar variable to see if the same happens.

https://github.com/Parallel-NetCDF/PnetCDF/blob/master/examples/C/nonblocking_write.c

Wei-keng

On Aug 7, 2023, at 4:28 PM, Jim Edwards <jedwards at ucar.edu> wrote:

Hi Wei-Keng,

The cb_nodes doesn't seem to be affected.

Not using independent mode doesn't seem to have helped.  I have the pioperf program now writing two files.  One with only
decomposed fields and one with one additional field, rundate, which is a string with the date in it.

The performance is drastically different:
                                                         IO tasks   vars      Mb/s                           Time (s)
 RESULT: write    SUBSET         1       256        64    12067.7548254854       25.1577347560    (without scalar)
 RESULT: write    SUBSET         1       256        64      286.4615089145     1059.8190875640      (with scalar)


On Mon, Aug 7, 2023 at 1:47 PM Wei-Keng Liao <wkliao at northwestern.edu<mailto:wkliao at northwestern.edu>> wrote:
Is that the reason for why cb_nodes is 1?
Strange, because cb_nodes is set at the file open time.

Entering the independent data mode in PnetCDF can be completely avoided
if using the nonblocking APIs.

I would suggest your codes to use the nonblocking APIs in the following way.

/* for non-partitioned variables */
if (rank == 0) {
    ncmpi_iput_var_int(fh, varid[0], data[0], &req[0]); /* write the whole variable */
    ncmpi_iput_var_int(fh, varid[1], data[1], &req[1]);
    ...
}
/* for partitioned variables */
ncmpi_iput_vara_int(fh, varid[j], data[j], starts[j], counts[j], &req[j]);
...


/* commit all posted nonblocking requests */
ncmpi_wait_all(ncid, NC_REQ_ALL, NC_REQ_NULL, NULL);


Wei-keng

> On Aug 7, 2023, at 2:12 PM, Jim Edwards <jedwards at ucar.edu<mailto:jedwards at ucar.edu>> wrote:
>
> Hi Wei-Keng,
>
> I think that I've found the problem.   In the model I am writing a number of scalar variables to the file as well as the decomposed variables.
> for the scalar variables I use a code structure like:
>
> ncmpi_begin_indep_data(fh);
> ncmpi_put_vars_int(fh, varid, start, count, stride, data);
> ncmpi_end_indep_data(fh);
>
> In my pioperf test code I didn't write any scalars - this morning I added one and the write performance for the decomposed variables got very very
> bad.  What can I do about it?
>
> Jim
>
>
> --
> Jim Edwards
>
> CESM Software Engineer
> National Center for Atmospheric Research
> Boulder, CO



--
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20230807/b806a59d/attachment-0001.html>