<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Barry, <br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof ContentPasted0" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I configured PETSC with --with-debugging=yes. I think this is enough to block any optimizations, right? </div>
<div class="elementToProof ContentPasted0" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I tried both Intel and GNU compilers. The error persists. </div>
<div class="elementToProof ContentPasted0" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I tried to change the preconditioner and use PILUT instead of BoomerAMG from Hypre. Still, the error appears. </div>
<div class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I noticed that I do not have any problem when only one ksp is present. I am afraid that this is the culprit. </div>
<div class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I will try your suggestion and setup an X Windows DISPLAY variable and send back the results.</div>
<div class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for your time, <br>
</div>
<div class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Pantelis<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>Από:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Στάλθηκε:</b> Πέμπτη, 9 Νοεμβρίου 2023 5:53 μμ<br>
<b>Προς:</b> Pantelis Moschopoulos <pmoschopoulos@outlook.com><br>
<b>Κοιν.:</b> petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Θέμα:</b> Re: [petsc-users] PETSC breaks when using HYPRE preconditioner</font>
<div> </div>
</div>
<div style="line-break:after-white-space">
<div> Pantelis</div>
<div><br>
</div>
If you can set an X Windows DISPLAY variable that works you can run with -on_error_attach_debugger and gdb should pop up in an Xterm on MPI rank 16 showing the code where it is crashing (based on Valgrind <span style="font-family:Times,"Times New Roman",serif; font-size:12pt">Address
0x0 is not stack'd, malloc'd or (recently) free'd </span>there will be pointer of 0 that should not be). Or if the computer system has some parallel debugger you can use that directly. For lldb use -on_error_attach_debugger lldb
<div><br>
</div>
<div> If you have some compiler optimizations set when you ./configure PETSc you might try making another PETSC_ARCH without optimizations (this is PETSc's default when you do not use --with-debugging=0). Does it still crash with no optimizations? Perhaps
also try with a different compiler?</div>
<div><br>
</div>
<div> Barry</div>
<div><br>
<div><br>
</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On Nov 9, 2023, at 6:57 AM, Pantelis Moschopoulos <pmoschopoulos@outlook.com> wrote:</div>
<br class="x_Apple-interchange-newline">
<div>
<div class="x_elementToProof" style="font-family:Helvetica; font-size:18px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<span style="font-family:Times,"Times New Roman",serif">Hello everyone,</span><br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span style="font-family:Times,"Times New Roman",serif">I am trying to use Petsc coupled with Hypre BoomerAMG as preconditioner in our in-house code to simulate the transient motion of complex fluid with finite elements. The problem is that after a random number
of iterations, an error arises when the Hypre is called.</span><br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span style="font-family:Times,"Times New Roman",serif">The error that I get in the terminal is the following: </span></div>
<div class="x_elementToProof x_ContentPasted0" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: ------------------------------------------------------------------------</span>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: or see<span class="x_Apple-converted-space"> </span><a href="https://petsc.org/release/faq/#valgrind">https://petsc.org/release/faq/#valgrind</a><span class="x_Apple-converted-space"> </span>and<span class="x_Apple-converted-space"> </span><a href="https://petsc.org/release/faq/">https://petsc.org/release/faq/</a></span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: --------------------- Stack Frames ------------------------------------</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: The line numbers in the error traceback are not always exact.</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #1 Hypre solve</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #2 PCApply_HYPRE() at /home/pmosx/Libraries/petsc/src/ksp/pc/impls/hypre/hypre.c:451</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #3 PCApply() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:486</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #4 PCApplyBAorAB() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:756</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #5 KSP_PCApplyBAorAB() at /home/pmosx/Libraries/petsc/include/petsc/private/kspimpl.h:443</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #6 KSPGMRESCycle() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:146</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #7 KSPSolve_GMRES() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:227</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #8 KSPSolve_Private() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:910</span></div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">[16]PETSC ERROR: #9 KSPSolve() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:1082</span></div>
<div class="x_ContentPasted0"><br>
</div>
<div class="x_ContentPasted0"><span style="font-family:Times,"Times New Roman",serif">On the same time, I use valgrind and when the program stops, it reports the following:</span><br>
</div>
<div class="x_ContentPasted0 x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== Invalid read of size 8</span>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== at 0x4841C74: _intel_fast_memcpy (in /usr/lib/x86_64-linux-gnu/valgrind/<a href="http://vgpreload_memcheck-amd64-linux.so/">vgpreload_memcheck-amd64-linux.so</a>)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x16231F73: hypre_GaussElimSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x1622DB4F: hypre_BoomerAMGCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x1620002E: hypre_BoomerAMGSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libhypre-2.29.0.so/">libHYPRE-2.29.0.so</a>)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x12B6F8F8: PCApply_HYPRE (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x12C38785: PCApply (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x12C36A39: PCApplyBAorAB (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x126299E1: KSPGMRESCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x12628051: KSPSolve_GMRES (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x127A532E: KSPSolve_Private (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x127A3C8A: KSPSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== by 0x12C50AF1: kspsolve_ (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/<a href="http://libpetsc.so/">libpetsc.so</a>.3.20.1)</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647== Address 0x0 is not stack'd, malloc'd or (recently) free'd</span></div>
<div class="x_ContentPasted1"><span style="font-family:Times,"Times New Roman",serif">==1261647==</span></div>
<div class="x_ContentPasted1"><br>
</div>
</div>
<div class="x_ContentPasted0 x_ContentPasted1 x_ContentPasted2"><span style="font-family:Times,"Times New Roman",serif; font-size:12pt">This is indeed a very peculiar error. I cannot understand why it happens. In our solution procedure, we split the equations
and we solve them segregated. I create two different ksp (1_ksp and 2_ksp) using<span class="x_Apple-converted-space"> </span></span><span style="font-family:Times,"Times New Roman",serif; font-size:12pt">KSPSetOptionsPrefix. Might this choice create a confusion
and results in this error?<br>
</span></div>
<br>
<div class="x_ContentPasted0 x_ContentPasted1 x_ContentPasted2"><span style="font-family:Times,"Times New Roman",serif; font-size:12pt">Any help is much appreciated.</span><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div class="x_ContentPasted0 x_ContentPasted1 x_ContentPasted2"><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div class="x_ContentPasted0 x_ContentPasted1 x_ContentPasted2"><span style="font-family:Times,"Times New Roman",serif; font-size:12pt">Pantelis</span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>