performance issue

[Wei-Keng Liao]

File striping size of 128 MB seems too big to me.
I have never tried anything larger than 16 MB and
learned that large sizes often performed worse.
striping_unit=134217728

Have you tried just 1 MB?

Wei-keng

On Aug 10, 2023, at 12:01 PM, Jim Edwards <jedwards at ucar.edu> wrote:



On Thu, Aug 10, 2023 at 10:53 AM Wei-Keng Liao <wkliao at northwestern.edu<mailto:wkliao at northwestern.edu>> wrote:
From the file header dump you sent earlier, no PnetCDF hint is necessary.

What MPI-IO hints are you using?
I am now using:
MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=48:striping_unit=134217728

we explored the parameter space and this seems to be about the best for this particular file.   But I think that there is still something
wrong at a low level and am working on trying to figure that out.


One question. Did you delete the output file before each run?

Yes



Wei-keng

On Aug 10, 2023, at 11:35 AM, Jim Edwards <jedwards at ucar.edu<mailto:jedwards at ucar.edu>> wrote:

We are having a lot of problems with darshan on our system - I'm working with the darshan developers to resolve.

I am setting MPIIO hints - but I see that there are some hints specific to pnetcdf - do you have any recommendations?

On Thu, Aug 10, 2023 at 10:15 AM Wei-Keng Liao <wkliao at northwestern.edu<mailto:wkliao at northwestern.edu>> wrote:
Hi, Jim

FYI. Darshan now can capture the I/O activities of PnetCDF,
in addition to the already MPI-IO and POSIX-IO.

Wei-keng

On Aug 9, 2023, at 6:22 PM, Wei-Keng Liao <wkliao at northwestern.edu<mailto:wkliao at northwestern.edu>> wrote:

In that case, I have the E3SM-IO benchmark that has a fairly complicate I/O
partitioning pattern. It used the decomposition maps generated from PIO.
https://github.com/Parallel-NetCDF/E3SM-IO<https://urldefense.com/v3/__https://github.com/Parallel-NetCDF/E3SM-IO__;!!Dq0X2DkFhyF93HkjWTBQKhk!UJAnknjDPQWY43szbAutlokjDrSHyBOcTlhEOChqYtThxnr812hxwvb8aSmHcEPiNOxNdOUHZi9Z2fyXbg2j6JH1HTo$>

Wei-keng

On Aug 9, 2023, at 6:17 PM, Jim Edwards <jedwards at ucar.edu<mailto:jedwards at ucar.edu>> wrote:

I think that your example case is too simple - it's doing a simple block decomposition.
In order to get the performance difference I am observing I need to do a more complicated
mapping.   I will work on a program that reproduces the problem without pio but it may take a
while.





--
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO



--
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/parallel-netcdf/attachments/20230810/da4240ab/attachment-0001.html>