[petsc-users] Meaning of Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end

[Barry Smith]

   Almost for sure the process has swallowed up all the system memory and so the batch system or os has killed the job. Likely you need to run on more nodes to solve this large a problem.

   Barry



> On Aug 9, 2016, at 2:24 AM, Athena Paz <athena.paz1 at gmail.com> wrote:
> 
> Hi all,
> 
> I'm very new to PETSC. I'm trying to solve a diffusion problem in 3D. I tried running a 500 x 500 x 500 grid using 20 processors but I encounter the following error:
> 
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:       INSTEAD the line number of the start of the function
> [0]PETSC ERROR:       is given.
> [0]PETSC ERROR: [0] MatSetValues_SeqAIJ line 441 /home/paz/petsc-3.7.3/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: [0] MatSetValues line 1157 /home/paz/petsc-3.7.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] MatSetValuesLocal line 2019 /home/paz/petsc-3.7.3/src/mat/interface/matrix.c
> [0]PETSC ERROR: [0] DMCreateMatrix_DA_3d_MPIAIJ line 1036 /home/paz/petsc-3.7.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: [0] DMCreateMatrix_DA line 625 /home/paz/petsc-3.7.3/src/dm/impls/da/fdda.c
> [0]PETSC ERROR: [0] DMCreateMatrix line 1171 /home/paz/petsc-3.7.3/src/dm/interface/dm.c
> [0]PETSC ERROR: [0] SNESSetUpMatrices line 579 /home/paz/petsc-3.7.3/src/snes/interface/snes.c
> [0]PETSC ERROR: [0] SNESSetUp_NEWTONLS line 303 /home/paz/petsc-3.7.3/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: [0] SNESSetUp line 2661 /home/paz/petsc-3.7.3/src/snes/interface/snes.c
> [0]PETSC ERROR: [0] SNESSolve line 3958 /home/paz/petsc-3.7.3/src/snes/interface/snes.c
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.7.3, Jul, 24, 2016
> [0]PETSC ERROR: ./ex7 on a arch-linux2-c-debug named akagi by paz Tue Aug  9 16:01:17 2016
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --download-mpich --with-debugging
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> [unset]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
>   What does this mean? I am able to run the code successfully with a 300x300x300 grid size. I also tried using -malloc_debug and valgrind as suggested in the Debugging FAQ for a small grid size and the code comes out clean. Any help is much appreciated!
> 
> 
> Thank you all for your time! Have a great day!
> 
> 
> Athena