<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Oct 17, 2013 at 5:30 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
On Oct 17, 2013, at 9:26 AM, Bishesh Khanal <<a href="mailto:bisheshkh@gmail.com">bisheshkh@gmail.com</a>> wrote:<br>
><br>
><br>
> --------------------------------------------------------------------------<br>
> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
> [0]PETSC ERROR: Caught signal number 15 Terminate: Somet process (or the batch system) has told this process to end<br>
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
> [0]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[0]PETSC</a> ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>
> [0]PETSC ERROR: likely location of problem given in stack below<br>
> [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
> [0]PETSC ERROR: INSTEAD the line number of the start of the function<br>
> [0]PETSC ERROR: is given.<br>
> [0]PETSC ERROR: [0] MatSetValues_MPIAIJ line 505 /tmp/petsc-3.4.3/src/mat/impls/aij/mpi/mpiaij.c<br>
> [0]PETSC ERROR: [0] MatSetValues line 1071 /tmp/petsc-3.4.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: [0] MatSetValuesLocal line 1935 /tmp/petsc-3.4.3/src/mat/interface/matrix.c<br>
> [0]PETSC ERROR: [0] DMCreateMatrix_DA_3d_MPIAIJ line 1051 /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c<br>
> [0]PETSC ERROR: [0] DMCreateMatrix_DA line 627 /tmp/petsc-3.4.3/src/dm/impls/da/fdda.c<br>
> [0]PETSC ERROR: [0] DMCreateMatrix line 900 /tmp/petsc-3.4.3/src/dm/interface/dm.c<br>
> [0]PETSC ERROR: [0] KSPSetUp line 192 /tmp/petsc-3.4.3/src/ksp/ksp/interface/itfunc.c<br>
> [0]PETSC ERROR: [0] solveModel line 122 "unknowndirectory/"/epi/asclepios2/bkhanal/works/AdLemModel/src/PetscAdLemTaras3D.cxx<br>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
> [0]PETSC ERROR: Signal received!<br>
> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
> [0]PETSC ERROR: Petsc Release Version 3.4.3, Oct, 15, 2013<br>
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.<br>
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.<br>
> [0]PETSC ERROR: See docs/index.html for manual pages.<br>
> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
> [0]PETSC ERROR: /epi/asclepios2/bkhanal/works/AdLemModel/build/src/AdLemMain on a arch-linux2-cxx-debug named nef002 by bkhanal Thu Oct 17 15:55:33 2013<br>
> [0]PETSC ERROR: Libraries linked from /epi/asclepios2/bkhanal/petscDebug/lib<br>
> [0]PETSC ERROR: Configure run at Wed Oct 16 14:18:48 2013<br>
> [0]PETSC ERROR: Configure options --with-mpi-dir=/opt/openmpi-gcc/current/ --with-shared-libraries --prefix=/epi/asclepios2/bkhanal/petscDebug -download-f-blas-lapack=1 --download-metis --download-parmetis --download-superlu_dist --download-scalapack --download-mumps --download-hypre --with-clanguage=cxx<br>
> [0]PETSC ERROR: ------------------------------------------------------------------------<br>
> [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file<br>
> ==47363==<br>
> ==47363== HEAP SUMMARY:<br>
> ==47363== in use at exit: 10,939,838,029 bytes in 8,091 blocks<br>
> ==47363== total heap usage: 1,936,963 allocs, 1,928,872 frees, 11,530,164,042 bytes allocated<br>
> ==47363==<br>
><br>
> Does it mean it is crashing near MatSetValues_MPIAIJ ?<br>
<br>
</div></div>> It is not really "crashing" here. The most likely cause is that the OS has run out of memory to provide so it has started shutting down processes; this is the "unfriendly" way that Unix handles "running out of memory". A much less likely possibility (since the program has just started) is that the process has run out of allocated time and the OS is shutting it down.<br>
<br>
I am confident in this case it is simply a mater of the system running out of memory. What happens if you run the exact same job without the valgrind? Please send the ENTIRE error message.<br></blockquote><div><br></div>
<div>Thanks!! Yes it seems that it was because of the running out of the memory. I realized that in the job script I submitted to the cluster I had limited the memory to a certain value. I deleted that line and resubmitted the job, now I the job gets killed because of the wall time (of 3 hours) being exceeded unlike the previous case where the mpiexec killed the job quite quickly. Now I will run the job again by increasing the wall time and see if it gives results or not and get back here. Thanks again!<br>
<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
<br>
<br>
<br>
<br>
</font></span></blockquote></div><br></div></div>