<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">Am 17.03.2021 um 15:15 schrieb Sanjoy Kumar Mazumder <<a href="mailto:mazumder@purdue.edu" class="">mazumder@purdue.edu</a>>:</div><br class="Apple-interchange-newline"><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Hi all,</span><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">I am trying to solve a set of coupled stiff ODEs in parallel using TSSUNDIALS with SUNDIALS_BDF as 'TSSundialsSetType' in PETSc. I am using a sparse Jacobian matrix of type MATMPIAIJ with no preconditioner. It runs for a long time with a very small timestep (~10^-8 - 10^-10) and then terminates abruptly with the following error:</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">'slurmstepd: error: Detected 4 oom-kill event(s) in step 1701844.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.'</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div></div></div></blockquote>This general class of problem can arise if there is a (small) memory leak occuring at every time step, so that is the first thing to rule out. </div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">After going through some of the common suggestions in the mailing list before,<span class="Apple-converted-space"> </span><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">1) I tried increasing the memory alloted per cpu (--mem-per-cpu) in my batch script but the problem still remains.<span class="Apple-converted-space"> </span><br class=""></div></div></div></blockquote><div class="">When you tried increasing the memory allocated per CPU, did the solver take more timesteps before the OOM error?</div><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">2) I have also checked for proper deallocation of the arrays in my function and jacobian sub-routines before every TS iteration.</div></div></div></blockquote>Did you confirm this with a tool like valgrind? If not, Is it possible for you to run a few time steps of your code on a local machine with valgrind?</div><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">3) The time allotted for my job in the assigned nodes (wall-time) far exceed the time for which the job is actually running.</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Is there anything I am missing out or not doing properly? Given below is the complete error that is showing up after the termination.</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Thanks</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">With regards,<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">Sanjoy<br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class=""><br class=""></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">--------------------------------------------------------------------------<div class="">Primary job terminated normally, but 1 process returned</div><div class="">a non-zero exit code. Per user-direction, the job has been aborted.</div><div class="">--------------------------------------------------------------------------</div><div class="">--------------------------------------------------------------------------</div><div class="">MPI_ABORT was invoked on rank 11 in communicator MPI_COMM_WORLD</div><div class="">with errorcode 50176059.</div><div class=""><br class=""></div><div class="">NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</div><div class="">You may or may not see output from other processes, depending on</div><div class="">exactly when Open MPI kills them.</div><div class="">--------------------------------------------------------------------------</div><div class="">[1]PETSC ERROR: ------------------------------------------------------------------------</div><div class="">[1]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div class="">[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div class="">[1]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div class="">[1]PETSC ERROR: or try<span class="Apple-converted-space"> </span><a href="http://valgrind.org/" class="">http://valgrind.org</a><span class="Apple-converted-space"> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div class="">[1]PETSC ERROR: likely location of problem given in stack below</div><div class="">[1]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div class="">[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div class="">[1]PETSC ERROR: INSTEAD the line number of the start of the function</div><div class="">[1]PETSC ERROR: is given.</div><div class="">[1]PETSC ERROR: [1] TSStep_Sundials line 121 /home/mazumder/petsc-3.14.5/src/ts/impls/implicit/sundials/sundials.c</div><div class="">[1]PETSC ERROR: [1] TSStep line 3736 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[1]PETSC ERROR: [1] TSSolve line 4046 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">[1]PETSC ERROR: Signal received</div><div class="">[1]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a><span class="Apple-converted-space"> </span>for trouble shooting.</div><div class="">[1]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021</div><div class="">[1]PETSC ERROR: ./ThO2 on a arch-linux-c-debug named<span class="Apple-converted-space"> </span><a href="http://bell-a017.rcac.purdue.edu/" class="">bell-a017.rcac.purdue.edu</a><span class="Apple-converted-space"> </span>by mazumder Mon Mar 15 13:26:36 2021</div><div class="">[1]PETSC ERROR: Configure options --with-cc-mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack --download-sundials=yes --with-debugging</div><div class="">[1]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div class="">[2]PETSC ERROR: ------------------------------------------------------------------------</div><div class="">[2]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div class="">[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div class="">[2]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div class="">[2]PETSC ERROR: or try<span class="Apple-converted-space"> </span><a href="http://valgrind.org/" class="">http://valgrind.org</a><span class="Apple-converted-space"> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div class="">[2]PETSC ERROR: likely location of problem given in stack below</div><div class="">[2]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div class="">[2]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div class="">[2]PETSC ERROR: INSTEAD the line number of the start of the function</div><div class="">[2]PETSC ERROR: is given.</div><div class="">[2]PETSC ERROR: [2] TSStep_Sundials line 121 /home/mazumder/petsc-3.14.5/src/ts/impls/implicit/sundials/sundials.c</div><div class="">[2]PETSC ERROR: [2] TSStep line 3736 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[2]PETSC ERROR: [2] TSSolve line 4046 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">[2]PETSC ERROR: Signal received</div><div class="">[2]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a><span class="Apple-converted-space"> </span>for trouble shooting.</div><div class="">[2]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021</div><div class="">[2]PETSC ERROR: ./ThO2 on a arch-linux-c-debug named<span class="Apple-converted-space"> </span><a href="http://bell-a017.rcac.purdue.edu/" class="">bell-a017.rcac.purdue.edu</a><span class="Apple-converted-space"> </span>by mazumder Mon Mar 15 13:26:36 2021</div><div class="">[2]PETSC ERROR: Configure options --with-cc-mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack --download-sundials=yes --with-debugging</div><div class="">[2]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div class="">[3]PETSC ERROR: ------------------------------------------------------------------------</div><div class="">[3]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div class="">[3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div class="">[3]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div class="">[3]PETSC ERROR: or try<span class="Apple-converted-space"> </span><a href="http://valgrind.org/" class="">http://valgrind.org</a><span class="Apple-converted-space"> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div class="">[3]PETSC ERROR: likely location of problem given in stack below</div><div class="">[3]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div class="">[3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div class="">[3]PETSC ERROR: INSTEAD the line number of the start of the function</div><div class="">[3]PETSC ERROR: is given.</div><div class="">[3]PETSC ERROR: [3] TSStep_Sundials line 121 /home/mazumder/petsc-3.14.5/src/ts/impls/implicit/sundials/sundials.c</div><div class="">[3]PETSC ERROR: [3] TSStep line 3736 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[3]PETSC ERROR: [3] TSSolve line 4046 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">[3]PETSC ERROR: Signal received</div><div class="">[3]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a><span class="Apple-converted-space"> </span>for trouble shooting.</div><div class="">[3]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021</div><div class="">[3]PETSC ERROR: ./ThO2 on a arch-linux-c-debug named<span class="Apple-converted-space"> </span><a href="http://bell-a017.rcac.purdue.edu/" class="">bell-a017.rcac.purdue.edu</a><span class="Apple-converted-space"> </span>by mazumder Mon Mar 15 13:26:36 2021</div><div class="">[3]PETSC ERROR: Configure options --with-cc-mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack --download-sundials=yes --with-debugging</div><div class="">[3]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div class="">[4]PETSC ERROR: ------------------------------------------------------------------------</div><div class="">[4]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end</div><div class="">[4]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</div><div class="">[4]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a></div><div class="">[4]PETSC ERROR: or try<span class="Apple-converted-space"> </span><a href="http://valgrind.org/" class="">http://valgrind.org</a><span class="Apple-converted-space"> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors</div><div class="">[4]PETSC ERROR: likely location of problem given in stack below</div><div class="">[4]PETSC ERROR: --------------------- Stack Frames ------------------------------------</div><div class="">[4]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,</div><div class="">[4]PETSC ERROR: INSTEAD the line number of the start of the function</div><div class="">[4]PETSC ERROR: is given.</div><div class="">[4]PETSC ERROR: [4] TSStep_Sundials line 121 /home/mazumder/petsc-3.14.5/src/ts/impls/implicit/sundials/sundials.c</div><div class="">[4]PETSC ERROR: [4] TSStep line 3736 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[4]PETSC ERROR: [4] TSSolve line 4046 /home/mazumder/petsc-3.14.5/src/ts/interface/ts.c</div><div class="">[4]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div><div class="">[4]PETSC ERROR: Signal received</div><div class="">[4]PETSC ERROR: See<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a><span class="Apple-converted-space"> </span>for trouble shooting.</div><div class="">[4]PETSC ERROR: Petsc Release Version 3.14.5, Mar 03, 2021</div><div class="">[4]PETSC ERROR: ./ThO2 on a arch-linux-c-debug named<span class="Apple-converted-space"> </span><a href="http://bell-a017.rcac.purdue.edu/" class="">bell-a017.rcac.purdue.edu</a><span class="Apple-converted-space"> </span>by mazumder Mon Mar 15 13:26:36 2021</div><div class="">[4]PETSC ERROR: Configure options --with-cc-mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack --download-sundials=yes --with-debugging</div>[4]PETSC ERROR: #1 User provided function() line 0 in unknown file</div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;" class="">--------------------------------------------------------------------------<div class="">mpirun noticed that process rank 0 with PID 0 on node bell-a017 exited on signal 9 (Killed).</div><div class="">--------------------------------------------------------------------------</div><div class="">[<a href="http://bell-a017.rcac.purdue.edu:62310/" class="">bell-a017.rcac.purdue.edu:62310</a>] 62 more processes have sent help message help-mpi-api.txt / mpi-abort</div><div class="">[<a href="http://bell-a017.rcac.purdue.edu:62310/" class="">bell-a017.rcac.purdue.edu:62310</a>] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</div>slurmstepd: error: Detected 4 oom-kill event(s) in step 1701844.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.</div></div></div></blockquote></div><br class=""></div></div></body></html>