[petsc-users] PETSC breaks when using HYPRE preconditioner

[Barry Smith]

  Pantelis

   If you can set an X Windows DISPLAY variable that works you can run with -on_error_attach_debugger and gdb should pop up in an Xterm on MPI rank 16 showing the code where it is crashing (based on Valgrind Address 0x0 is not stack'd, malloc'd or (recently) free'd there will be pointer of 0 that should not be). Or if the computer system has some parallel debugger you can use that directly. For lldb use -on_error_attach_debugger lldb

  If you have some compiler optimizations set when you ./configure PETSc you might try making another PETSC_ARCH without optimizations (this is PETSc's default when you do not use --with-debugging=0). Does it still crash with no optimizations?  Perhaps also try with a different compiler?

  Barry




> On Nov 9, 2023, at 6:57 AM, Pantelis Moschopoulos <pmoschopoulos at outlook.com> wrote:
> 
> Hello everyone,
> 
> I am trying to use Petsc coupled with Hypre BoomerAMG as preconditioner in our in-house code to simulate the transient motion of complex fluid with finite elements. The problem is that after a random number of iterations, an error arises when the Hypre is called.
> 
> The error that I get in the terminal is the following: 
> [16]PETSC ERROR: ------------------------------------------------------------------------
> [16]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [16]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [16]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind and https://petsc.org/release/faq/
> [16]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [16]PETSC ERROR: The line numbers in the error traceback are not always exact.
> [16]PETSC ERROR: #1 Hypre solve
> [16]PETSC ERROR: #2 PCApply_HYPRE() at /home/pmosx/Libraries/petsc/src/ksp/pc/impls/hypre/hypre.c:451
> [16]PETSC ERROR: #3 PCApply() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:486
> [16]PETSC ERROR: #4 PCApplyBAorAB() at /home/pmosx/Libraries/petsc/src/ksp/pc/interface/precon.c:756
> [16]PETSC ERROR: #5 KSP_PCApplyBAorAB() at /home/pmosx/Libraries/petsc/include/petsc/private/kspimpl.h:443
> [16]PETSC ERROR: #6 KSPGMRESCycle() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:146
> [16]PETSC ERROR: #7 KSPSolve_GMRES() at /home/pmosx/Libraries/petsc/src/ksp/ksp/impls/gmres/gmres.c:227
> [16]PETSC ERROR: #8 KSPSolve_Private() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:910
> [16]PETSC ERROR: #9 KSPSolve() at /home/pmosx/Libraries/petsc/src/ksp/ksp/interface/itfunc.c:1082
> 
> On the same time, I use valgrind and when the program stops, it reports the following:
> ==1261647== Invalid read of size 8
> ==1261647==    at 0x4841C74: _intel_fast_memcpy (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so <http://vgpreload_memcheck-amd64-linux.so/>)
> ==1261647==    by 0x16231F73: hypre_GaussElimSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so <http://libhypre-2.29.0.so/>)
> ==1261647==    by 0x1622DB4F: hypre_BoomerAMGCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so <http://libhypre-2.29.0.so/>)
> ==1261647==    by 0x1620002E: hypre_BoomerAMGSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libHYPRE-2.29.0.so <http://libhypre-2.29.0.so/>)
> ==1261647==    by 0x12B6F8F8: PCApply_HYPRE (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x12C38785: PCApply (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x12C36A39: PCApplyBAorAB (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x126299E1: KSPGMRESCycle (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x12628051: KSPSolve_GMRES (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x127A532E: KSPSolve_Private (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x127A3C8A: KSPSolve (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==    by 0x12C50AF1: kspsolve_ (in /home/pmosx/Libraries/PETSC_INS_DIR_INTELDebug/lib/libpetsc.so <http://libpetsc.so/>.3.20.1)
> ==1261647==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> ==1261647==
> 
> This is indeed a very peculiar error. I cannot understand why it happens. In our solution procedure, we split the equations and we solve them segregated. I create two different ksp (1_ksp and 2_ksp) using KSPSetOptionsPrefix. Might this choice create a confusion and results in this error?
> 
> Any help is much appreciated.
> 
> Pantelis

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20231109/c6e3fdb1/attachment.html>