<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div> Pantelis</div><div><br></div> If you can set an X Windows DISPLAY variable that works you can run with -on_error_attach_debugger and gdb should pop up in an Xterm on MPI rank 16 showing the code where it is crashing (based on Valgrind <span style="font-family: Times, "Times New Roman", serif; font-size: 12pt;">Address 0x0 is not stack'd, malloc'd or (recently) free'd </span>there will be pointer of 0 that should not be). Or if the computer system has some parallel debugger you can use that directly. For lldb use -on_error_attach_debugger lldb<div><br></div><div> If you have some compiler optimizations set when you ./configure PETSc you might try making another PETSC_ARCH without optimizations (this is PETSc's default when you do not use --with-debugging=0). Does it still crash with no optimizations? Perhaps also try with a different compiler?</div><div><br></div><div> Barry</div><div><br><div><br></div><div><br><div><br><blockquote type="cite"><div>On Nov 9, 2023, at 6:57 AM, Pantelis Moschopoulos <pmoschopoulos@outlook.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="font-family: Times, "Times New Roman", serif;">Hello everyone,</span><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Times, "Times New Roman", serif;">I am trying to use Petsc coupled with Hypre BoomerAMG as preconditioner in our in-house code to simulate the transient motion of complex fluid with finite elements. The problem is that after a random number of iterations, an error arises when the Hypre is called.</span><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Times, "Times New Roman", serif;">The error that I get in the terminal is the following: </span></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: ------------------------------------------------------------------------</span><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://petsc.org/release/faq/#valgrind">https://petsc.org/release/faq/#valgrind</a><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span><a href="https://petsc.org/release/faq/">https://petsc.org/release/faq/</a></span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: --------------------- Stack Frames ------------------------------------</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: The line numbers in the error traceback are not always exact.</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #1 Hypre solve</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #2 PCApply_HYPRE() at /home/pmosx/Libraries/petsc/src/ksp/pc/impls/hypre/hypre.c:451</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #3 PCApply() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:486</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #4 PCApplyBAorAB() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:756</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #5 KSP_PCApplyBAorAB() at /home/pmosx/Libraries/petsc/include/petsc/private/kspimpl.h:443</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #6 KSPGMRESCycle() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:146</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #7 KSPSolve_GMRES() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:227</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #8 KSPSolve_Private() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:910</span></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">[16]PETSC ERROR: #9 KSPSolve() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:1082</span></div><div class="ContentPasted0"><br></div><div class="ContentPasted0"><span style="font-family: Times, "Times New Roman", serif;">On the same time, I use valgrind and when the program stops, it reports the following:</span><br></div><div class="ContentPasted0 ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== Invalid read of size 8</span><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== at 0x4841C74: _intel_fast_memcpy (in /usr/lib/x86_64-linux-gnu/valgrind/<a href="http://vgpreload_memcheck-amd64-linux.so/">vgpreload_memcheck-amd64-linux.so</a>)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x16231F73: hypre_GaussElimSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x1622DB4F: hypre_BoomerAMGCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x1620002E: hypre_BoomerAMGSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x12B6F8F8: PCApply_HYPRE (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x12C38785: PCApply (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x12C36A39: PCApplyBAorAB (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x126299E1: KSPGMRESCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x12628051: KSPSolve_GMRES (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x127A532E: KSPSolve_Private (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x127A3C8A: KSPSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== by 0x12C50AF1: kspsolve_ (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647== Address 0x0 is not stack'd, malloc'd or (recently) free'd</span></div><div class="ContentPasted1"><span style="font-family: Times, "Times New Roman", serif;">==1261647==</span></div><div class="ContentPasted1"><br></div></div><div class="ContentPasted0 ContentPasted1 ContentPasted2"><span style="font-family: Times, "Times New Roman", serif; font-size: 12pt;">This is indeed a very peculiar error. I cannot understand why it happens. In our solution procedure, we split the equations and we solve them segregated. I create two different ksp (1_ksp and 2_ksp) using<span class="Apple-converted-space"> </span></span><span style="font-family: Times, "Times New Roman", serif; font-size: 12pt;">KSPSetOptionsPrefix. Might this choice create a confusion and results in this error?<br></span></div><br><div class="ContentPasted0 ContentPasted1 ContentPasted2"><span style="font-family: Times, "Times New Roman", serif; font-size: 12pt;">Any help is much appreciated.</span><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div class="ContentPasted0 ContentPasted1 ContentPasted2"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></span></div><div class="ContentPasted0 ContentPasted1 ContentPasted2"><span style="font-family: Times, "Times New Roman", serif; font-size: 12pt;">Pantelis</span></div></div></div></blockquote></div><br></div></div></body></html>