Hi Pravind,<br><br>Thought I'd try MPI-IO that you mentioned so I went to my favourite MPI site <a href="https://computing.llnl.gov/tutorials/mpi/">https://computing.llnl.gov/tutorials/mpi/</a> and there was no mention of it. There's also no links or documentation on MPICH2 home page so that might explain why I missed it :)<br>
<br>I've found some documentation and may give it a quick try but I think I've also fixed my problem.<br><br>I had a bash script to monitor the progress of the MPI job that went like:<br><br>mpiexec .... >report.file&<br>
while ..<br>do<br> echo `date` `wc -l report.file`<br> sleep 30<br>done<br><br>If I monitor file size rather than number of lines using stat -c%s rather than wc -l then I don't get the slow downs and everything runs perfectly.<br>
<br>It's interesting because server has a lot of ram and the file is in cache. When I try wc -l on 2G file (maximum size it gets to) it takes about 1sec and monitoring loop sleeps for 30 secs between each wc -l but it obviously interferes with mpiexec and hydra_pmi_proxy.<br>
<br>So I'm happy now and thanks for your help. Just knowing mpiexec & hydra_pmi_proxy CPU was related to stdout IO was enough of a clue to get me moving in the right direction<br><br>Thanks, Colin<br><br><br><div class="gmail_quote">
On Fri, Jan 21, 2011 at 8:34 AM, Colin Hercus <span dir="ltr"><<a href="mailto:colin@novocraft.com">colin@novocraft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Pravind,<br><br> That's 1Gb of stdout from master process only in in ~ 1hr is that really a lot? Writes are in 16K blocks. I also read about the same using stdio routines. I'll take a look at MPI-I/O<br><font color="#888888"><br>
Colin</font><div><div></div><div class="h5"><br>
<br><div class="gmail_quote">On Fri, Jan 21, 2011 at 2:53 AM, Pavan Balaji <span dir="ltr"><<a href="mailto:balaji@mcs.anl.gov" target="_blank">balaji@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><br>
On 01/20/2011 09:51 AM, Colin Hercus wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
The 1Gb is MPI messages, about the same is written to stdout.<br>
</blockquote>
<br></div>
1GB of stdout might cause the proxy and mpiexec to be busy for a long time moving the data. Shouldn't you be considering MPI-I/O if you are writing such large amounts?<div><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
My PC for compile and (static) link was at 1.2.1 and I was running on a<br>
server with 1.3.1 when I first noticed this. I then upgraded to 1.3.1<br>
but it made no difference.<br>
</blockquote>
<br></div>
Yes, mpiexec is backward compatible to any version of MPICH2 (or any other derivative of MPICH2). But it's best to always upgrade to the latest version.<div><br>
<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
When would mpiexec and hydra_pmi_proxy use CPU? is it just related to<br>
stdout?<br>
</blockquote>
<br></div>
Mostly, only for stdout/stderr/stdin + when one MPI process needs to know the address of another remote MPI process (typically when you send the first message).<div><div></div><div><br>
<br>
-- Pavan<br>
<br>
-- <br>
Pavan Balaji<br>
<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>