Hi Matt,<br><br>The serial job is running without any problems and never stalls. Actually the parallel jobs also running successfully on distributed-memory desktop or on single node of cluster. It will get stuck if it is running on more than one compute node(now it is running on two nodes). Both the serial job and parallel job (running on distributed or cluster) I mentioned before have the same size(dofs). But If I ran a smaller job on cluster with two nodes, it might not get stuck and work fine. <br>
<br>As you said before, I add MAT_ASSEMBLY_FLUSH after every element stiffness matrix is inserted. I got the output like below, and it gets stuck too.<br><br>[0] MatStashScatterBegin_Private()<div id=":w8">: No of messages: 1 <br>
[0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 24584 <br>[0] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>
[3] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>[2] MatAssemblyBegin_MPIAIJ(): Stash has 4096 entries, uses 0 mallocs.<br>[7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>[5] MatAssemblyBegin_MPIAIJ(): Stash has 2048 entries, uses 0 mallocs.<br>
[4] MatAssemblyBegin_MPIAIJ(): Stash has 2048 entries, uses 0 mallocs.<br>[6] MatAssemblyBegin_MPIAIJ(): Stash has 1024 entries, uses 0 mallocs.<br>[1] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>
[0] MatStashScatterBegin_Private(): No of messages: 1 <br>[0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 24584 <br>[0] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>[3] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>
[2] MatAssemblyBegin_MPIAIJ(): Stash has 4096 entries, uses 0 mallocs.<br>[7] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs.<br>[4] MatAssemblyBegin_MPIAIJ(): Stash has 2048 entries, uses 0 mallocs.<br>[5] MatAssemblyBegin_MPIAIJ(): Stash has 2048 entries, uses 0 mallocs.<br>
[6] MatAssemblyBegin_MPIAIJ(): Stash has 1024 entries, uses 0 mallocs.<br>[1] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>[0] MatStashScatterBegin_Private(): No of messages: 1 <br>[0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 24584 <br>
[0] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>[3] MatAssemblyBegin_MPIAIJ(): Stash has 3072 entries, uses 0 mallocs.<br>[2] MatAssemblyBegin_MPIAIJ(): Stash has 4096 entries, uses 0 mallocs.</div>
<br><br>On Fri, Jan 20, 2012 at 10:21 AM, Wen Jiang <<a href="mailto:jiangwen84@gmail.com">jiangwen84@gmail.com</a>> wrote:<br>
<br>
> Hi, Matt<br>
><br>
> Could you tell me some more details about how to get a stack trace there?<br>
> I know little about it. The job is submitted on head node and running on<br>
> compute nodes.<br>
><br>
<br>
1) Always run serial problems until you understand what is happening<br>
<br>
2) Run with -start_in_debugger, and type 'cont' in the debugger (read about<br>
gdb)<br>
<br>
3) When it stalls, Ctrl-C and then type 'where'<br>
<br>
Matt<br>
<br>