Hi,<br><br>This is my first post here and I apologise if this has been discussed before but I keep getting timeouts while trying to access the archives.<br><br>I have a fairly simple MPICH2 job where master process reads a transaction files and send them to the slaves for processing. It also collects output from the slaves and writes it to stdout.<br>
<br>The other day I noticed that the whole process seems to slow to a crawl with hydra_pmi_proxy using 100% CPU, mpiexec popping up on top at 50-100% CPU and my slaves and master recording almost no CPU time. Sometimes this clears itself and the process speeds up again, sometimes not.<br>
<br>I figured this is probably my problem so I spent all week trying to fix it (but I don't know why it happens so I'm not really getting anywhere.<br><br>I tried ch3:nem and ch3:sock with no difference<br><br>My slave processes are multi-threaded but don't do any MPI calls from the threads, and I can run with just a single thread.<br>
<br>My standard run on 4 servers is mpiexec -np 5 so I have one master task with a single thread and 4 slaves each multithreaded to the number of CPUs on the server. This is a little over committed but master usually uses about 1% CPU so it's not much.<br>
<br>I then tried<br><br>mpiexec -np 49 with single threaded slaves (I have 48 cores)<br>mpiexec -np 48<br>mpiexec -np 44 <br>mpiexec -np 32<br>mpiexec -np 24<br><br>all these exhibit the same problem.<br><br>My slaves are very simple<br>
<br>while() {<br> MPI_Probe( 0, MPI_ANY_TAG, <br> MPI_Get_Count...<br> if(tag == EOF) break;<br> if(not first) MPI_SendSz(...)<br> MPI_Recv()<br>}<br>MPI_SendSz(...)<br><br>The master process is a tad more complicated and I've rewritten it several times in attempts to fix it.<br>
<br>Original<br><br>for each slave task<br> MPI_Issend( first message)<br><br>while(..) {<br> MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG... for receiving messages<br> if(flag) {<br> MPI_Recv<br> fwrite(stdout, buffer)<br>
}<br> MPI_TestAny (Issend Requests)<br> if(flag)<br> MPI_Issend(next buffer)<br>}<br><br>This loop was using Probe Any Source Any Tag and I read somewhere this could be a problem so I rewrote the above loop <br>
to use MPI_Irecv. In the first attempt I used two MPI_request arrays, one for sends and one for receives and the main loop had to TestAnys.<br><br>I then changed to a single array of requests where Reads were first part of the array and sends the second. This allowed me to use WaitAny so now my master code looks like:<br>
<br>for each slave task {<br>
MPI_Issend( first message)<br> MPI_Irecv()<br>}<br>
<br>
while(..) {<br>
MPI_WaitAny(....)<br>
if(idx != MPI_UNDEFINED) {<br> if(idx < num_slaves) {<br>
fwrite(stdout, buffer)<br>
MPI_Irecv<br>
} else<br>
MPI_Issend(next buffer)<br> }<br>
}<br>
<br>These changes haven't made the slightest difference.<br><br>A few more details:<br><br>Message buffer is 16Kbytes ( I've also tried 200byte, 2K &4K buffers)<br><br>Problem doesn't occur until the program has been running for about an hour. In that hour there would be about 1Gbyte of messages. It then occurs intermittently during the remainder of the run.<br>
<br>Running exactly same settings and data again and the slow down will occur at a different time in the run.<br><br>Servers are on a 1G Ethernet LAN<br><br>Disk files are on same server as the master so no other network traffic.<br>
<br>I have exclusive use of servers for this test, only top and some other job monitoring that uses minimal CPU is running.<br><br>So I'm stuck, not sure what to try next and would appreciate any suggestions or help.<br>
<br>Thanks, Colin<br>