<div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">I tried the GPFS on derecho again with the fix and</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">RESULT: write SUBSET 1 16 64 5450.6689151051 3.4050866580</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">RESULT: write SUBSET 1 16 64 5953.4351587908 3.1175278650</div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:#38761d">So maybe there is a measurable difference on GPFS - it's just particularly hard on lustre.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 14, 2023 at 1:29 PM Jim Edwards <<a href="mailto:jedwards@ucar.edu">jedwards@ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">Yes, the same test on lustre and GPFS. <br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">derecho:<br></div><div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)"><p><font color="#38761d">GPFS (/glade/work on derecho):</font><br>
<font color="#38761d">RESULT: write SUBSET 1 16 64 4570.2078677815 4.0610844270</font><br>
<font color="#38761d">RESULT: write SUBSET 1 16 64 4470.3231494386 4.1518251320</font><font color="#38761d"></font></p>
<p><font color="#38761d">Lustre, default PFL's:</font><br>
<font color="#38761d">RESULT: write SUBSET 1 16 64 2808.6570137094 6.6081404420</font><br>
<font color="#38761d">RESULT: write SUBSET 1 16 64 1025.1671656858 18.1043644600</font></p><p><font color="#38761d"><font color="#38761d">LUSTRE, no PFL's lfs setstripe -c 48 -S 128<br>
RESULT: write SUBSET 1 16 64 4687.6852437580 3.9593102000</font><br>
<font color="#38761d"> RESULT: write SUBSET 1 16 64 3001.4741125579 6.1836282120</font></font></p><p>perlmutter, <font color="#38761d"><font color="#38761d">no PFL's lfs setstripe -c 48 -S 128</font></font></p><p>RESULT: write SUBSET 1 16 64 5174.0815629926 <span> </span><a href="tel:35871100550" rel="noopener noreferrer" style="box-sizing:inherit;text-decoration:none" target="_blank">3.5871100550</a><br></p><p>RESULT: write SUBSET 1 16 64 3176.2693942192 <span> </span><a href="tel:58433330730" rel="noopener noreferrer" style="box-sizing:inherit;text-decoration:none" target="_blank">5.843333073</a><span role="presentation" style="box-sizing:inherit;display:inline-flex;height:fit-content;width:fit-content"><button aria-hidden="true" aria-label="View Ben Kirk’s Profile" type="button" style="box-sizing:inherit;background:rgba(0,0,0,0) none repeat scroll 0% 0%;border:0px none;padding:0px;font:inherit;color:inherit;margin:0px;overflow:initial;text-align:initial;vertical-align:initial;display:inline-block;outline:currentcolor none medium;height:36px;width:36px"></button></span></p><p>Frontera (uses impi not cray-mpich)<br></p><pre style="box-sizing:inherit;margin:4px 0px;padding:8px;font-size:12px;font-variant-ligatures:none;line-height:1.50001;white-space:pre-wrap;word-break:normal;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;border-radius:4px;overflow-y:hidden;color:rgb(29,28,29);font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">RESULT: write SUBSET 1 36 64 243.4676204728 75.0407794043
RESULT: write SUBSET 1 36 64 30.8135655567 592.9206721112<br><br></pre><pre style="box-sizing:inherit;margin:4px 0px;padding:8px;font-size:12px;font-variant-ligatures:none;line-height:1.50001;white-space:pre-wrap;word-break:normal;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;border-radius:4px;overflow-y:hidden;color:rgb(29,28,29);font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">impi can turn off lustre optimizations with env variables:<br><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;white-space:normal;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">unset I_MPI_EXTRA_FILESYSTEM_FORCE<br><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;white-space:normal;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">unset I_MPI_EXTRA_FILESYSTEM<br>RESULT: write SUBSET 1 36 64 315.4529589578 57.9167177901
<br>RESULT: write SUBSET 1 36 64 303.0899778031 60.2791294269<br><br></span></span></pre><pre style="box-sizing:inherit;margin:4px 0px;padding:8px;font-size:12px;font-variant-ligatures:none;line-height:1.50001;white-space:pre-wrap;word-break:normal;font-family:Monaco,Menlo,Consolas,"Courier New",monospace;border-radius:4px;overflow-y:hidden;color:rgb(29,28,29);font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;white-space:normal;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none"><span style="color:rgb(29,28,29);font-family:Slack-Lato,Slack-Fractions,appleLogo,sans-serif;font-size:15px;font-style:normal;font-variant-ligatures:common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:left;text-indent:0px;text-transform:none;word-spacing:0px;white-space:normal;background-color:rgb(248,248,248);text-decoration-style:initial;text-decoration-color:initial;display:inline;float:none">GPFS cheyenne:<br> RESULT: write SUBSET 1 16 64 6126.5760973514 3.0294245440 <br> RESULT: write SUBSET 1 16 64 4638.4045534969 4.0013758580</span></span></pre><p></p><p><br></p><p><br></p></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 14, 2023 at 1:11 PM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Did you run the same tests on a non-Lustre file system and see no difference?</div>
Can you show me the timings?
<div><br>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 14, 2023, at 11:54 AM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-Keng,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Thanks for looking into this. Because the allocations in pioperformance.F90 are done on the compute nodes and</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
not the IO nodes I don't think that your suggestion would make any difference. I also wonder why this issue
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
appears to be so specific to the lustre file system - presumably the ROMIO functionality you speak of is general and not</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
specific to lustre? Anyway your analysis spurred me to try something else which seems to work: prior to calling</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
ncmpi_iput_varn in pio_darray_int.c I added a call to ncmpi_wait_all to make sure that any existing buffer was written.</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
This seems to have fixed the problem and my writes are now</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
RESULT: write SUBSET 1 16 64 4787.6393342631 3.8766495770<br>
RESULT: write SUBSET 1 16 64 4803.9296372205 3.8635037150<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Aug 14, 2023 at 9:47 AM Wei-Keng Liao <<a href="mailto:wkliao@northwestern.edu" target="_blank">wkliao@northwestern.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="auto">
<div dir="auto">
<div dir="auto">
<div dir="auto">
<div dir="auto">
<div dir="auto">Hi, Jim
<div><br>
</div>
<div>Digging into ROMIO source codes, I found the root cause of the timing</div>
<div>difference between the two test cases is whether or not the user buffer</div>
<div>passed to MPI_File_write_all is contiguous.</div>
<div><br>
</div>
<div>In your test program, the write buffers for all record variables are</div>
<div>allocated in a <span>contiguous space, while the</span> fix-sized variable is in a</div>
<div>separate memory space.</div>
<div><a href="https://urldefense.com/v3/__https://github.com/jedwards4b/ParallelIO/blob/25b471d5864db1cf7b8dfa26bd5d568eceba1a04/tests/performance/pioperformance.F90*L220-L227__;Iw!!Dq0X2DkFhyF93HkjWTBQKhk!TtiUA1y3rS-K1Ci1HJaI5-nAJVx4QmQHf1GQLtAtYnyfITDBJ9tfJc2Ckg-D6o4KnMowEJ-fG-V_LHZ69baTrKA$" target="_blank">https://github.com/jedwards4b/ParallelIO/blob/25b471d5864db1cf7b8dfa26bd5d568eceba1a04/tests/performance/pioperformance.F90#L220-L227</a></div>
<div><br>
</div>
<div><span><br>
</span></div>
<div><span>Therefore, in case of writing an extra fix-sized variable, the </span><span>aggregated</span></div>
<div><span>write buffer is noncontiguous, while in the other case </span><span>contiguous</span><span>.</span></div>
<div><br>
</div>
<div>When the write buffer is not contiguous, ROMIO allocates an internal buffer,</div>
<div>copies the data over, and uses it to perform communication. When the buffer</div>
<div>is contiguous, ROMIO uses the user buffer <span>directly for communication</span>.</div>
<div>Such coping can become expensive when the write amount is large.</div>
<div><br>
</div>
<div>If you want to verify this of my finding, please try allocating the</div>
<div>buffers of individual record variables separately. Let me know how it goes.</div>
<div>
<div><br>
</div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 6:03 PM, Jim Edwards <<a href="mailto:jedwards@ucar.edu" target="_blank">jedwards@ucar.edu</a>> wrote:</div>
<br>
<div>
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-keng,,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
For this case I'm using a RoundRobin distribution as shown here.<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
if(doftype .eq. 'ROUNDROBIN') then<br>
do i=1,varsize<br>
compmap(i) = (i-1)*npe+mype+1<br>
enddo</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div><div>Jim Edwards<br><br></div><font size="1">CESM Software Engineer<br></font></div><font size="1">National Center for Atmospheric Research<br></font></div><font size="1">Boulder, CO</font> <br></div></div>