Hi Pavan,<br><br>First apologies for calling you Pravind in earlier emails. Pravind is a past colleague and I was in automatic mode :)<br><br>I've been looking at MPI IO and I have a couple of questions:<br>1. Is it possible to use MPI IO to write to stdout? I couldn't see how to do this.<br>
<br>2. If I don't use MPI IO and stick to writing to stdout with fwrite() is it better to do a) Send from slave to master and then fwrite in master or b) write to stdout in the slaves.<br><br>Thanks, Colin<br><br><div class="gmail_quote">
On Fri, Jan 21, 2011 at 2:05 PM, Colin Hercus <span dir="ltr"><<a href="mailto:colin@novocraft.com">colin@novocraft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Pravind,<br><br>Thought I'd try MPI-IO that you mentioned so I went to my favourite MPI site <a href="https://computing.llnl.gov/tutorials/mpi/" target="_blank">https://computing.llnl.gov/tutorials/mpi/</a> and there was no mention of it. There's also no links or documentation on MPICH2 home page so that might explain why I missed it :)<br>
<br>I've found some documentation and may give it a quick try but I think I've also fixed my problem.<br><br>I had a bash script to monitor the progress of the MPI job that went like:<br><br>mpiexec .... >report.file&<br>
while ..<br>do<br> echo `date` `wc -l report.file`<br> sleep 30<br>done<br><br>If I monitor file size rather than number of lines using stat -c%s rather than wc -l then I don't get the slow downs and everything runs perfectly.<br>
<br>It's interesting because server has a lot of ram and the file is in cache. When I try wc -l on 2G file (maximum size it gets to) it takes about 1sec and monitoring loop sleeps for 30 secs between each wc -l but it obviously interferes with mpiexec and hydra_pmi_proxy.<br>
<br>So I'm happy now and thanks for your help. Just knowing mpiexec & hydra_pmi_proxy CPU was related to stdout IO was enough of a clue to get me moving in the right direction<br><br>Thanks, Colin<div><div></div>
<div class="h5"><br><br><br><div class="gmail_quote">
On Fri, Jan 21, 2011 at 8:34 AM, Colin Hercus <span dir="ltr"><<a href="mailto:colin@novocraft.com" target="_blank">colin@novocraft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Pravind,<br><br> That's 1Gb of stdout from master process only in in ~ 1hr is that really a lot? Writes are in 16K blocks. I also read about the same using stdio routines. I'll take a look at MPI-I/O<br><font color="#888888"><br>
Colin</font><div><div></div><div><br>
<br><div class="gmail_quote">On Fri, Jan 21, 2011 at 2:53 AM, Pavan Balaji <span dir="ltr"><<a href="mailto:balaji@mcs.anl.gov" target="_blank">balaji@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><br>
On 01/20/2011 09:51 AM, Colin Hercus wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
The 1Gb is MPI messages, about the same is written to stdout.<br>
</blockquote>
<br></div>
1GB of stdout might cause the proxy and mpiexec to be busy for a long time moving the data. Shouldn't you be considering MPI-I/O if you are writing such large amounts?<div><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
My PC for compile and (static) link was at 1.2.1 and I was running on a<br>
server with 1.3.1 when I first noticed this. I then upgraded to 1.3.1<br>
but it made no difference.<br>
</blockquote>
<br></div>
Yes, mpiexec is backward compatible to any version of MPICH2 (or any other derivative of MPICH2). But it's best to always upgrade to the latest version.<div><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
When would mpiexec and hydra_pmi_proxy use CPU? is it just related to<br>
stdout?<br>
</blockquote>
<br></div>
Mostly, only for stdout/stderr/stdin + when one MPI process needs to know the address of another remote MPI process (typically when you send the first message).<div><div></div><div><br>
<br>
-- Pavan<br>
<br>
-- <br>
Pavan Balaji<br>
<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>