<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
File striping size of 128 MB seems too big to me.
<div>I have never tried anything larger than 16 MB and</div>
<div>learned that large sizes often performed worse.
<div>
<blockquote type="cite" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class="gmail_quote">striping_unit=134217728</div>
</blockquote>
<div><br>
</div>
<div>Have you tried just 1 MB?</div>
<br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 10, 2023, at 12:01 PM, Jim Edwards <jedwards@ucar.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div><br class="Apple-interchange-newline">
<br style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at 10:53 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div>From the file header dump you sent earlier, no PnetCDF hint is necessary.
<div><br>
</div>
<div>What MPI-IO hints are you using?</div>
<div>
<div></div>
</div>
</div>
</blockquote>
<div><span class="gmail_default" style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">I am now using:</span></div>
<div>MPICH_MPIIO_HINTS=*:romio_cb_read=enable:romio_cb_write=enable:striping_factor=48:striping_unit=134217728</div>
<div><br>
</div>
<div>
<div class="gmail_default" style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">
we explored the parameter space and this seems to be about the best for this particular file. But I think that there is still something</div>
<div class="gmail_default" style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">
wrong at a low level and am working on trying to figure that out. <span class="Apple-converted-space"> </span><br>
</div>
<br>
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div>
<div><br>
</div>
<div>One question. Did you delete the output file before each run?<br>
<div><br>
</div>
</div>
</div>
</blockquote>
<div>
<div class="gmail_default" style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">
Yes</div>
<br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div>
<div>
<div></div>
<div>Wei-keng<span class="Apple-converted-space"> </span></div>
<div><br>
<blockquote type="cite">
<div>On Aug 10, 2023, at 11:35 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">We are having a lot of problems with darshan on our system - I'm working with the darshan developers to resolve.</div>
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);"><br>
</div>
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">I am setting MPIIO hints - but I see that there are some hints specific to pnetcdf - do you have any recommendations?<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 10, 2023 at 10:15 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div>Hi, Jim
<div><br>
</div>
<div>FYI. Darshan now can capture the I/O activities of PnetCDF,</div>
<div>in addition to the already MPI-IO and POSIX-IO.</div>
<div><br>
<div>Wei-keng<span class="Apple-converted-space"> </span></div>
<div><br>
<blockquote type="cite">
<div>On Aug 9, 2023, at 6:22 PM, Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:</div>
<br>
<div>
<div>In that case, I have the E3SM-IO benchmark that has a fairly complicate I/O
<div>partitioning pattern. It used the decomposition maps generated from PIO.</div>
<div><a href="https://urldefense.com/v3/__https://github.com/Parallel-NetCDF/E3SM-IO__;!!Dq0X2DkFhyF93HkjWTBQKhk!UJAnknjDPQWY43szbAutlokjDrSHyBOcTlhEOChqYtThxnr812hxwvb8aSmHcEPiNOxNdOUHZi9Z2fyXbg2j6JH1HTo$" target="_blank">https://github.com/Parallel-NetCDF/E3SM-IO</a><br>
<div><br>
<div>Wei-keng<span class="Apple-converted-space"> </span></div>
<div><br>
<blockquote type="cite">
<div>On Aug 9, 2023, at 6:17 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">I think that your example case is too simple - it's doing a simple block decomposition.</div>
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">In order to get the performance difference I am observing I need to do a more complicated</div>
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">mapping. I will work on a program that reproduces the problem without pio but it may take a<span class="Apple-converted-space"> </span><br>
</div>
<div style="font-family: "comic sans ms", sans-serif; color: rgb(56, 118, 29);">while. <span class="Apple-converted-space"> </span><br>
</div>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">--<span class="Apple-converted-space"> </span></span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font><span class="Apple-converted-space"> </span><br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all" style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<br style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<span class="gmail_signature_prefix" style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div dir="ltr" class="gmail_signature" style="caret-color: rgb(0, 0, 0); font-family: Monaco; font-size: 14px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>