<div dir="ltr"><div class="gmail_quote">On Tue, Feb 7, 2012 at 08:34, Derek Gaston <span dir="ltr"><<a href="mailto:friedmud@gmail.com">friedmud@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On Mon, Feb 6, 2012 at 10:27 PM, Jed Brown <span dir="ltr"><<a href="mailto:jedbrown@mcs.anl.gov" target="_blank">jedbrown@mcs.anl.gov</a>></span> wrote:<br></div><div class="gmail_quote"><div class="im">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_quote"><div><br></div><div>Are _all_ the processes making it here?</div></div></div></blockquote><div><br></div></div><div>Sigh. I knew someone was going to ask that ;-)</div><div><br>
</div><div>I'll have to write a short script to grab the stack trace from every one of the 10,000 processes to see where they are and try to find any anomalies. Anyone have a script (or pieces of one) to do this that they wouldn't mind sharing?</div>
<div><br></div><div>I did spot check quite a few and they were all in the same spot.</div><div><br></div><div>Now here comes the weirdness: I left one of these processes attached in GDB for quite a while (10+ minutes) after the whole job had been hung for over an hour. When I noticed that I had left it attached I detached GDB and.... the job started right up! That is: it moved on past this problem! How is that for some weirdness. It might have just been coincidence... or maybe me stalling that process for a bit by attaching GDB nudged some communication in the right direction... I don't know.</div>
</div></blockquote><div><br></div><div>Hmm, progress semantics of MPI should ensure completion. Stalling the process with gdb should not change anything (assuming you weren't actually making changes with gdb). Can you run with MPICH2?</div>
</div></div>