<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi, Jim
<div><br>
</div>
<div>Digging into ROMIO source codes, I found the root cause of the timing</div>
<div>difference between the two test cases is whether or not the user buffer</div>
<div>passed to MPI_File_write_all is contiguous.</div>
<div><br>
</div>
<div>In your test program, the write buffers for all record variables are</div>
<div>allocated in a <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">contiguous space, while the</span> fix-sized variable is in a</div>
<div>separate memory space.</div>
<div><a href="https://github.com/jedwards4b/ParallelIO/blob/25b471d5864db1cf7b8dfa26bd5d568eceba1a04/tests/performance/pioperformance.F90#L220-L227">https://github.com/jedwards4b/ParallelIO/blob/25b471d5864db1cf7b8dfa26bd5d568eceba1a04/tests/performance/pioperformance.F90#L220-L227</a></div>
<div><br>
</div>
<div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br>
</span></div>
<div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Therefore, in case of writing an extra fix-sized variable, the </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">aggregated</span></div>
<div><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">write buffer is noncontiguous, while in the other case </span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">contiguous</span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">.</span></div>
<div><br>
</div>
<div>When the write buffer is not contiguous, ROMIO allocates an internal buffer,</div>
<div>copies the data over, and uses it to perform communication. When the buffer</div>
<div>is contiguous, ROMIO uses the user buffer <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">directly for communication</span>.</div>
<div>Such coping can become expensive when the write amount is large.</div>
<div><br>
</div>
<div>If you want to verify this of my finding, please try allocating the</div>
<div>buffers of individual record variables separately. Let me know how it goes.</div>
<div>
<div><br>
</div>
<div>Wei-keng </div>
<div><br>
<blockquote type="cite">
<div>On Aug 11, 2023, at 6:03 PM, Jim Edwards <jedwards@ucar.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
Hi Wei-keng,,</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
For this case I'm using a RoundRobin distribution as shown here.<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
if(doftype .eq. 'ROUNDROBIN') then<br>
do i=1,varsize<br>
compmap(i) = (i-1)*npe+mype+1<br>
enddo</div>
<div class="gmail_default" style="font-family:comic sans ms,sans-serif;color:rgb(56,118,29)">
<br>
</div>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div>
<div>Jim Edwards<br>
<br>
</div>
<font size="1">CESM Software Engineer<br>
</font></div>
<font size="1">National Center for Atmospheric Research<br>
</font></div>
<font size="1">Boulder, CO</font> <br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>