<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">
<div class="moz-cite-prefix"><b><font size="-1">On 04/17/2018
08:04 AM, Carl Ponder wrote:</font></b><br>
</div>
<blockquote type="cite"
cite="mid:acb4033a-b8a5-57e3-a358-17149fe0eeed@nvidia.com">
<blockquote>
<p>my sysadmin's tell me that NFS is all that we're ever going
to have on our cluster.<br>
I can live with substandard I/O performance, I just want the
various I/O libraries to give the right data.<br>
We can try following these ROMIO instructions, but can you
answer some questions about it?<br>
</p>
<p>1) It says that ROMIO is used in MPICH. Just to confirm, do
MVAPICH2 & OpenMPI & Intel/MPI all use ROMIO?<br>
And even if not, would the instructions still be
relevant to these other MPI's?<br>
</p>
<p> 2) It talks about NFS version 3. It looks like we have a
combination of 3 & 4 on our system.<br>
Are the instructions relevant to NFS version 4?<br>
</p>
</blockquote>
<blockquote>
<p>3) Given the reservations about performance, I suppose I
could ask for a special directory to be created &
mounted for "coherent" operations, and leave the other
mounts as-is.<br>
Do you see any problems with doing this?</p>
</blockquote>
</blockquote>
<font size="-1"><b>
On 04/17/2018 08:15 AM, Latham, Robert J. wrote:</b></font>
<blockquote type="cite">
<blockquote cite="mid:1523970920.28306.9.camel@mcs.anl.gov">
<pre wrap="">I would do the following:
NFS is not a parallel file system. Read-only from NFS might be
OK. Writing is trickier because NFS implements bizzare caching
rules. Sometimes the clients hold onto data for a long time, even
after you ask every way possible to plase write this data out to disk.</pre>
</blockquote>
</blockquote>
Robert -- You mean that setting "noac" doesn't help, or that it
isn't a full fix?
<blockquote type="cite">
<blockquote cite="mid:1523970920.28306.9.camel@mcs.anl.gov">
<pre wrap="">I would use collective I/O, but then tell the MPI-IO library to carry
out all I/O from just one rank. For ROMIO if you set the hint
"cb_nodes" to "1", then all clients will use a single aggregator to
write, and you can hopefully sidestep any caching problems.
You should also set "romio_cb_write" to "enable" to force all writes,
even ones ROMIO might think are nice friendly large contiguous
requests, to take the two-phase collective buffering path.</pre>
</blockquote>
</blockquote>
I'll try to find a way to set these in the MPI environment
modules:<br>
<blockquote><tt>export cb_nodes=1</tt><tt><br>
</tt><tt>export romio_cb_write=enable</tt><br>
</blockquote>
<br>
</div>
<DIV>
<HR>
</DIV>
<DIV>This email message is for the sole use of the intended recipient(s) and may
contain confidential information. Any unauthorized review, use, disclosure
or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original
message. </DIV>
<DIV>
<HR>
</DIV>
</body>
</html>