performance issue

[Wei-Keng Liao]

Can you run command "ncoffsets -sg file.nc" that shows the sizes of file header
and all variables? For the cesm case, is _enddef called only once?

Could you also point me to the program files that call PnetCDF APIs, including
writing attributes and variables?


Wei-keng

On Aug 4, 2023, at 11:05 AM, Jim Edwards <jedwards at ucar.edu> wrote:

I am using the new ncar system, derecho<https://urldefense.com/v3/__https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwigtcytrsOAAxWXHjQIHVoDD6YQFnoECBcQAQ&url=https*3A*2F*2Farc.ucar.edu*2Fknowledge_base*2F74317833&usg=AOvVaw2aXlWuOfLnua7fFmIgvfoV&opi=89978449__;JSUlJSU!!Dq0X2DkFhyF93HkjWTBQKhk!Xq6u5krREolkIRHG8AL2taDCmg6HsEdgcEoviUVyzqUINi-ipPM1EhtMcJkQfUYghDhutn7DfH5Wjm57wJ9lQhc$>, which has a lustre parallel file system.

Looking at the difference between the two headers below makes me wonder if the issue is with variable attributes?


snip


On Fri, Aug 4, 2023 at 9:39 AM Wei-Keng Liao <wkliao at northwestern.edu<mailto:wkliao at northwestern.edu>> wrote:
Hi, Jim

Can your provide the test program and the file header dumped by "ncdump -h", if that is available?
Also, what machine was used in the tests and its the parallel file system configuration is?
These can help diagnose.

Wei-keng

On Aug 4, 2023, at 8:49 AM, Jim Edwards <jedwards at ucar.edu<mailto:jedwards at ucar.edu>> wrote:

I am using ncmpi_iput_varn and ncmpi_wait_all to write output from my model.   I have a test program that does nothing but test the
performance of the write operation.   Attached is a plot of performance in the model and in the standalone application.   I'm looking for
clues as to why the model performance is scaling so badly with the number of variables but the standalone program performance is fine.



--
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO
<Screenshot 2023-07-27 at 11.49.03 AM.png>



--
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20230804/5e0b161f/attachment.html>