<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Mark,<div class=""><br class=""></div><div class=""> Can you run in valgrind? </div><div class=""><br class=""></div><div class=""> Exactly what BLAS are you using? </div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 24, 2020, at 7:54 AM, Mark Lohry <<a href="mailto:mlohry@gmail.com" class="">mlohry@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Reran with debug mode and got a stack trace for this bus error, looks like it's happening in BLASgemv, see pasted below. I did take care of the ISColoring leak mentioned previously, although that was a very small amount of data and I don't think is relevant here.<br class=""></div><div class=""><br class=""></div><div class="">At this point it's happily run 222 timesteps prior to this, so I'm a little mystified. Any ideas?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">222 TS dt 0.03 time 6.66<br class=""> 0 SNES Function norm 4.124287265556e+02 <br class=""> 0 KSP Residual norm 4.124287265556e+02 <br class=""> 1 KSP Residual norm 4.123248052318e+02 <br class=""> 2 KSP Residual norm 4.123173350456e+02 <br class=""> 3 KSP Residual norm 4.118769044110e+02 <br class=""> 4 KSP Residual norm 4.094856150740e+02 <br class=""> 5 KSP Residual norm 4.006000788078e+02 <br class=""> 6 KSP Residual norm 3.787922969183e+02 <br class=""></div>[clip]<br class=""><div class=""><div class=""> Linear solve converged due to CONVERGED_RTOL iterations 9<br class=""> Line search: Using full step: fnorm 4.015236590684e+01 gnorm 3.173434863784e+00<br class=""> 2 SNES Function norm 3.173434863784e+00 <br class=""> Nonlinear solve converged due to CONVERGED_FNORM_RELATIVE iterations 2<br class=""> 0 SNES Function norm 5.842010710080e+02 <br class=""> 0 KSP Residual norm 5.842010710080e+02 <br class=""> 1 KSP Residual norm 5.840526408234e+02 <br class=""> 2 KSP Residual norm 5.840431857354e+02 <br class=""> 3 KSP Residual norm 5.834351392302e+02 <br class=""> 4 KSP Residual norm 5.800901047861e+02 <br class=""> 5 KSP Residual norm 5.675562288567e+02 <br class=""> 6 KSP Residual norm 5.366287895681e+02 <br class=""> 7 KSP Residual norm 4.725811521866e+02 <br class="">[911]PETSC ERROR: ------------------------------------------------------------------------<br class="">[911]PETSC ERROR: Caught signal number 7 BUS: Bus Error, possibly illegal memory access<br class="">[911]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br class="">[911]PETSC ERROR: or see <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br class="">[911]PETSC ERROR: or try <a href="http://valgrind.org/" class="">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br class="">[911]PETSC ERROR: likely location of problem given in stack below<br class="">[911]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br class="">[911]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br class="">[911]PETSC ERROR: INSTEAD the line number of the start of the function<br class="">[911]PETSC ERROR: is given.<br class="">[911]PETSC ERROR: [911] BLASgemv line 1393 /home/mlohry/build/external/petsc/src/mat/impls/baij/seq/baijfact.c<br class="">[911]PETSC ERROR: [911] MatSolve_SeqBAIJ_N_NaturalOrdering line 1378 /home/mlohry/build/external/petsc/src/mat/impls/baij/seq/baijfact.c<br class=""></div><div class="">[911]PETSC ERROR: [911] MatSolve line 3354 /home/mlohry/build/external/petsc/src/mat/interface/matrix.c<br class="">[911]PETSC ERROR: [911] PCApply_ILU line 201 /home/mlohry/build/external/petsc/src/ksp/pc/impls/factor/ilu/ilu.c<br class="">[911]PETSC ERROR: [911] PCApply line 426 /home/mlohry/build/external/petsc/src/ksp/pc/interface/precon.c<br class="">[911]PETSC ERROR: [911] KSP_PCApply line 279 /home/mlohry/build/external/petsc/include/petsc/private/kspimpl.h<br class="">[911]PETSC ERROR: [911] KSPSolve_PREONLY line 16 /home/mlohry/build/external/petsc/src/ksp/ksp/impls/preonly/preonly.c<br class="">[911]PETSC ERROR: [911] KSPSolve_Private line 590 /home/mlohry/build/external/petsc/src/ksp/ksp/interface/itfunc.c<br class="">[911]PETSC ERROR: [911] KSPSolve line 848 /home/mlohry/build/external/petsc/src/ksp/ksp/interface/itfunc.c<br class="">[911]PETSC ERROR: [911] PCApply_ASM line 441 /home/mlohry/build/external/petsc/src/ksp/pc/impls/asm/asm.c<br class="">[911]PETSC ERROR: [911] PCApply line 426 /home/mlohry/build/external/petsc/src/ksp/pc/interface/precon.c<br class="">[911]PETSC ERROR: [911] KSP_PCApply line 279 /home/mlohry/build/external/petsc/include/petsc/private/kspimpl.h<br class="">[911]PETSC ERROR: [911] KSPFGMRESCycle line 108 /home/mlohry/build/external/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c<br class="">[911]PETSC ERROR: [911] KSPSolve_FGMRES line 274 /home/mlohry/build/external/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c<br class="">[911]PETSC ERROR: [911] KSPSolve_Private line 590 /home/mlohry/build/external/petsc/src/ksp/ksp/interface/itfunc.c<br class="">[911]PETSC ERROR: [911] KSPSolve line 848 /home/mlohry/build/external/petsc/src/ksp/ksp/interface/itfunc.c<br class="">[911]PETSC ERROR: [911] SNESSolve_NEWTONLS line 144 /home/mlohry/build/external/petsc/src/snes/impls/ls/ls.c<br class="">[911]PETSC ERROR: [911] SNESSolve line 4403 /home/mlohry/build/external/petsc/src/snes/interface/snes.c<br class="">[911]PETSC ERROR: [911] TSStep_ARKIMEX line 728 /home/mlohry/build/external/petsc/src/ts/impls/arkimex/arkimex.c<br class="">[911]PETSC ERROR: [911] TSStep line 3682 /home/mlohry/build/external/petsc/src/ts/interface/ts.c<br class="">[911]PETSC ERROR: [911] TSSolve line 4005 /home/mlohry/build/external/petsc/src/ts/interface/ts.c<br class="">[911]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">[911]PETSC ERROR: Signal received<br class="">[911]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class="">[911]PETSC ERROR: Petsc Release Version 3.13.3, Jul 01, 2020 <br class="">[911]PETSC ERROR: maDG on a arch-linux2-c-opt named tiger-h20c2n20 by mlohry Sun Aug 23 19:54:21 2020<br class="">[911]PETSC ERROR: Configure options PETSC_DIR=/home/mlohry/build/external/petsc PETSC_ARCH=arch-linux2-c-opt --with-cc=/usr/local/openmpi/3.1.3/gcc/x8<br class="">[911]PETSC ERROR: #1 User provided function() line 0 in unknown file<br class="">--------------------------------------------------------------------------<br class="">MPI_ABORT was invoked on rank 911 in communicator MPI_COMM_WORLD<br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 12, 2020 at 8:19 PM Mark Lohry <<a href="mailto:mlohry@gmail.com" class="">mlohry@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""></div><div class=""> Perhaps you are calling ISColoringGetIS() and not calling ISColoringRestoreIS()? <br class=""></div></blockquote><div class=""><br class=""></div><div class="">I have matching ISColoringGet/Restore here, and it's only used prior to the first iteration so at least it doesn't seem to be growing. At the bottom I pasted the malloc_view and malloc_debug output from running 1 time step.</div><div class=""><br class=""></div><div class="">I'm sort of thinking this might be a red herring -- is it possible the rank 0 process is chewing up dramatically more memory than others, like with logging or something? Like I mentioned earlier the total memory usage is well under the machine limits. I'll spring in some PetscMemoryGetMaximumUsage logging at every time step and try to get a big job going again.<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""> Are you using Fortran? <br class=""></div></blockquote><div class=""><br class=""></div><div class="">C++ <br class=""></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class="">[ 0]1408 bytes PetscSplitReductionCreate() line 63 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c<br class="">[ 0]80 bytes PetscSplitReductionCreate() line 57 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c<br class="">[ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c<br class="">[ 0]16 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]272 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]880 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]960 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]976 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1040 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]64 bytes ISColoringGetIS() line 266 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c<br class="">[ 0]32 bytes PetscCommDuplicate() line 129 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c<br class="">[0] Maximum memory PetscMalloc()ed 610153776 maximum size of entire process 719073280<br class="">[0] Memory usage sorted by function<br class="">[0] 6 192 DMCoarsenHookAdd()<br class="">[0] 2 9984 DMCreate()<br class="">[0] 2 128 DMCreate_Shell()<br class="">[0] 2 64 DMDSEnlarge_Static()<br class="">[0] 1 672 DMKSPCreate()<br class="">[0] 3 96 DMRefineHookAdd()<br class="">[0] 3 2064 DMSNESCreate()<br class="">[0] 4 128 DMSubDomainHookAdd()<br class="">[0] 1 768 DMTSCreate()<br class="">[0] 2 96 ISColoringCreate()<br class="">[0] 8 12608 ISColoringGetIS()<br class="">[0] 1 307200 ISConcatenate()<br class="">[0] 29 25984 ISCreate()<br class="">[0] 25 400 ISCreate_General()<br class="">[0] 4 64 ISCreate_Stride()<br class="">[0] 20 338016 ISGeneralSetIndices_General()<br class="">[0] 3 921600 ISGetIndices_Stride()<br class="">[0] 2 307232 ISGlobalToLocalMappingSetUp_Basic()<br class="">[0] 1 6144 ISInvertPermutation_General()<br class="">[0] 3 308576 ISLocalToGlobalMappingCreate()<br class="">[0] 2 32 KSPConvergedDefaultCreate()<br class="">[0] 2 2816 KSPCreate()<br class="">[0] 1 224 KSPCreate_FGMRES()<br class="">[0] 1 8016 KSPGMRESClassicalGramSchmidtOrthogonalization()<br class="">[0] 2 16032 KSPSetUp_FGMRES()<br class="">[0] 4 16084160 KSPSetUp_GMRES()<br class="">[0] 2 36864 MatColoringApply_SL()<br class="">[0] 1 656 MatColoringCreate()<br class="">[0] 6 17088 MatCreate()<br class="">[0] 1 16 MatCreateMFFD_WP()<br class="">[0] 1 16 MatCreateSubMatrices_SeqBAIJ()<br class="">[0] 1 12288 MatCreateSubMatrix_SeqBAIJ()<br class="">[0] 3 32320 MatCreateSubMatrix_SeqBAIJ_Private()<br class="">[0] 2 1472 MatCreate_MFFD()<br class="">[0] 1 416 MatCreate_SeqAIJ()<br class="">[0] 3 864 MatCreate_SeqBAIJ()<br class="">[0] 2 416 MatCreate_Shell()<br class="">[0] 1 784 MatFDColoringCreate()<br class="">[0] 2 12288 MatFDColoringDegreeSequence_Minpack()<br class="">[0] 6 30859392 MatFDColoringSetUp_SeqXAIJ()<br class="">[0] 3 42512 MatGetColumnIJ_SeqAIJ()<br class="">[0] 4 72720 MatGetColumnIJ_SeqBAIJ_Color()<br class="">[0] 1 6144 MatGetOrdering_Natural()<br class="">[0] 2 36384 MatGetRowIJ_SeqAIJ()<br class="">[0] 7 210626000 MatILUFactorSymbolic_SeqBAIJ()<br class="">[0] 2 313376 MatIncreaseOverlap_SeqBAIJ()<br class="">[0] 2 30740608 MatLUFactorNumeric_SeqBAIJ_N()<br class="">[0] 1 6144 MatMarkDiagonal_SeqAIJ()<br class="">[0] 1 6144 MatMarkDiagonal_SeqBAIJ()<br class="">[0] 8 256 MatRegisterRootName()<br class="">[0] 1 6160 MatSeqAIJCheckInode()<br class="">[0] 4 115216 MatSeqAIJSetPreallocation_SeqAIJ()<br class="">[0] 4 302779424 MatSeqBAIJSetPreallocation_SeqBAIJ()<br class="">[0] 13 576 MatSolverTypeRegister()<br class="">[0] 1 16 PCASMCreateSubdomains()<br class="">[0] 2 1664 PCCreate()<br class="">[0] 1 160 PCCreate_ASM()<br class="">[0] 1 192 PCCreate_ILU()<br class="">[0] 5 307264 PCSetUp_ASM()<br class="">[0] 2 416 PetscBTCreate()<br class="">[0] 2 3216 PetscClassPerfLogCreate()<br class="">[0] 2 1616 PetscClassRegLogCreate()<br class="">[0] 2 32 PetscCommBuildTwoSided_Allreduce()<br class="">[0] 2 64 PetscCommDuplicate()<br class="">[0] 2 1888 PetscDSCreate()<br class="">[0] 2 26416 PetscEventPerfLogCreate()<br class="">[0] 2 158400 PetscEventPerfLogEnsureSize()<br class="">[0] 2 1616 PetscEventRegLogCreate()<br class="">[0] 2 9600 PetscEventRegLogRegister()<br class="">[0] 8 102400 PetscFreeSpaceGet()<br class="">[0] 474 15168 PetscFunctionListAdd_Private()<br class="">[0] 2 528 PetscIntStackCreate()<br class="">[0] 142 11360 PetscLayoutCreate()<br class="">[0] 56 896 PetscLayoutSetUp()<br class="">[0] 59 9440 PetscObjectComposedDataIncreaseReal()<br class="">[0] 2 576 PetscObjectListAdd()<br class="">[0] 33 768 PetscOptionsGetEList()<br class="">[0] 1 16 PetscOptionsHelpPrintedCreate()<br class="">[0] 1 32 PetscPushSignalHandler()<br class="">[0] 7 6944 PetscSFCreate()<br class="">[0] 3 432 PetscSFCreate_Basic()<br class="">[0] 2 1472 PetscSFLinkCreate()<br class="">[0] 11 1229040 PetscSFSetUpRanks()<br class="">[0] 7 614512 PetscSFSetUp_Basic()<br class="">[0] 4 20096 PetscSegBufferCreate()<br class="">[0] 2 1488 PetscSplitReductionCreate()<br class="">[0] 2 3008 PetscStageLogCreate()<br class="">[0] 1148 23872 PetscStrallocpy()<br class="">[0] 6 13056 PetscStrreplace()<br class="">[0] 9 3456 PetscTableCreate()<br class="">[0] 1 16 PetscViewerASCIIOpen()<br class="">[0] 6 96 PetscViewerAndFormatCreate()<br class="">[0] 1 752 PetscViewerCreate()<br class="">[0] 1 96 PetscViewerCreate_ASCII()<br class="">[0] 2 1424 SNESCreate()<br class="">[0] 1 16 SNESCreate_NEWTONLS()<br class="">[0] 1 1008 SNESLineSearchCreate()<br class="">[0] 1 16 SNESLineSearchCreate_BT()<br class="">[0] 16 1824 SNESMSRegister()<br class="">[0] 46 9056 TSARKIMEXRegister()<br class="">[0] 1 1264 TSAdaptCreate()<br class="">[0] 8 384 TSBasicSymplecticRegister()<br class="">[0] 1 2160 TSCreate()<br class="">[0] 1 224 TSCreate_Theta()<br class="">[0] 48 5968 TSGLEERegister()<br class="">[0] 41 7728 TSRKRegister()<br class="">[0] 89 14736 TSRosWRegister()<br class="">[0] 71 110192 VecCreate()<br class="">[0] 1 307200 VecCreateGhostWithArray()<br class="">[0] 123 36874080 VecCreate_MPI_Private()<br class="">[0] 7 4300800 VecCreate_Seq()<br class="">[0] 8 256 VecCreate_Seq_Private()<br class="">[0] 6 400 VecDuplicateVecs_Default()<br class="">[0] 3 2352 VecScatterCreate()<br class="">[0] 7 1843296 VecScatterSetUp_SF()<br class="">[0] 126 2016 VecStashCreate_Private()<br class="">[0] 1 3072 mapBlockColoringToJacobian()<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 12, 2020 at 4:22 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Yes, there are some PETSc objects or arrays that you are not freeing so they are printed at the end of the run. For small runs this harmless but if new objects/memory is allocated at each iteration and not suitably freed it will eventually add up.<div class=""><br class=""></div><div class=""> Run with -malloc_view (small problem with say 2 iterations) it will print everything allocated and might be helpful.<br class=""><div class=""><br class=""></div><div class=""> Perhaps you are calling ISColoringGetIS() and not calling ISColoringRestoreIS()? </div><div class=""><br class=""></div><div class=""> It is also possible it is a leak in PETSc, but that is unlikely since we test for them.</div><div class=""><br class=""></div><div class=""> Are you using Fortran? </div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 12, 2020, at 1:29 PM, Mark Lohry <<a href="mailto:mlohry@gmail.com" target="_blank" class="">mlohry@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">Thanks Matt and Barry. At Matt's suggestion I ran a smaller representative case with valgrind and didn't see anything alarming (apart from a small leak in an older boost version I was using: <a href="https://github.com/boostorg/serialization/issues/104" target="_blank" class="">https://github.com/boostorg/serialization/issues/104</a> although I don't think this was causing the issue).</div><div class=""><br class=""></div><div class="">-malloc_debug dumps quite a lot, this is supposed to be empty right? Output pasted below. It looks like the same sequence of calls is repeated 8 times, which is how many nonlinear solves occurred in this particular run. Thoughts?<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">[ 0]1408 bytes PetscSplitReductionCreate() line 63 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c<br class="">[ 0]80 bytes PetscSplitReductionCreate() line 57 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c<br class="">[ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c<br class="">[ 0]16 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]272 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]880 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]960 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]976 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]1040 bytes ISGeneralSetIndices_General() line 578 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]16 bytes PetscLayoutSetUp() line 269 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]80 bytes PetscLayoutCreate() line 55 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c<br class="">[ 0]16 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]32 bytes PetscStrallocpy() line 187 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c<br class="">[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c<br class="">[ 0]16 bytes ISCreate_General() line 647 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c<br class="">[ 0]896 bytes ISCreate() line 37 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c<br class="">[ 0]64 bytes ISColoringGetIS() line 266 in /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c<br class="">[ 0]32 bytes PetscCommDuplicate() line 129 in /home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 12, 2020 at 1:46 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Mark.<div class=""><br class=""></div><div class=""> When valgrind is not feasible (like on many centrally controlled batch systems) you can run PETSc with an extra flag to do some memory error checks</div><div class=""> -malloc_debug</div><div class=""><br class=""></div><div class=""> this </div><div class=""><br class=""></div><div class="">1) fills all malloced memory with Nan so if the code is using uninitialized memory it may be detected and </div><div class="">2) checks the beginning and end of each alloced memory region for out-of-bounds writes at each malloc and free.</div><div class=""><br class=""></div><div class="">it will slow the code down a little bit but generally not a huge amount.</div><div class=""><br class=""></div><div class="">It is no where near as good as valgrind or other memory corruption tools but it has the advantage you can run it anywhere on any size job.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 12, 2020, at 7:46 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Wed, Aug 12, 2020 at 7:53 AM Mark Lohry <<a href="mailto:mlohry@gmail.com" target="_blank" class="">mlohry@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">I'm getting seemingly random failures of late:</div><div class="">Caught signal number 7 BUS: Bus Error, possibly illegal memory access</div></div></blockquote><div class=""><br class=""></div><div class="">The first thing I would do is run valgrind on as wide an array of tests as you can. This will find problems</div><div class="">on things that run completely fine.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Symptoms:</div><div class="">1) Seems to only happen (so far) on larger cases, 400-2000 cores</div><div class="">2) It doesn't happen right away -- this was running happily for several hours over several hundred time steps with no indication of bad health in the numerics</div><div class="">3) At least the total memory consumption seems to be within bounds, though I'm not sure about individual processes. e.g. slurm here reported Memory Efficiency: 75.23% of 1.76 TB (180.00 GB/node)</div><div class="">4) running the same setup twice it fails at different points<br class=""></div><div class=""><br class=""></div><div class="">Any suggestions on what to look for? This is a bit painful to work on as I can only reproduce it on large runs and then it's seemingly random.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark<br class=""></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>