[petsc-users] Bus Error
Mark Lohry
mlohry at gmail.com
Wed Aug 12 19:19:19 CDT 2020
>
> Perhaps you are calling ISColoringGetIS() and not calling
> ISColoringRestoreIS()?
>
I have matching ISColoringGet/Restore here, and it's only used prior to the
first iteration so at least it doesn't seem to be growing. At the bottom I
pasted the malloc_view and malloc_debug output from running 1 time step.
I'm sort of thinking this might be a red herring -- is it possible the rank
0 process is chewing up dramatically more memory than others, like with
logging or something? Like I mentioned earlier the total memory usage is
well under the machine limits. I'll spring in some
PetscMemoryGetMaximumUsage logging at every time step and try to get a big
job going again.
Are you using Fortran?
>
C++
[ 0]1408 bytes PetscSplitReductionCreate() line 63 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c
[ 0]80 bytes PetscSplitReductionCreate() line 57 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c
[ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c
[ 0]16 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]272 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]880 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]960 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]976 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]1024 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]1040 bytes ISGeneralSetIndices_General() line 578 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]16 bytes PetscLayoutSetUp() line 269 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]80 bytes PetscLayoutCreate() line 55 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
[ 0]16 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]32 bytes PetscStrallocpy() line 187 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
[ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
[ 0]16 bytes ISCreate_General() line 647 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
[ 0]896 bytes ISCreate() line 37 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
[ 0]64 bytes ISColoringGetIS() line 266 in
/home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c
[ 0]32 bytes PetscCommDuplicate() line 129 in
/home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c
[0] Maximum memory PetscMalloc()ed 610153776 maximum size of entire process
719073280
[0] Memory usage sorted by function
[0] 6 192 DMCoarsenHookAdd()
[0] 2 9984 DMCreate()
[0] 2 128 DMCreate_Shell()
[0] 2 64 DMDSEnlarge_Static()
[0] 1 672 DMKSPCreate()
[0] 3 96 DMRefineHookAdd()
[0] 3 2064 DMSNESCreate()
[0] 4 128 DMSubDomainHookAdd()
[0] 1 768 DMTSCreate()
[0] 2 96 ISColoringCreate()
[0] 8 12608 ISColoringGetIS()
[0] 1 307200 ISConcatenate()
[0] 29 25984 ISCreate()
[0] 25 400 ISCreate_General()
[0] 4 64 ISCreate_Stride()
[0] 20 338016 ISGeneralSetIndices_General()
[0] 3 921600 ISGetIndices_Stride()
[0] 2 307232 ISGlobalToLocalMappingSetUp_Basic()
[0] 1 6144 ISInvertPermutation_General()
[0] 3 308576 ISLocalToGlobalMappingCreate()
[0] 2 32 KSPConvergedDefaultCreate()
[0] 2 2816 KSPCreate()
[0] 1 224 KSPCreate_FGMRES()
[0] 1 8016 KSPGMRESClassicalGramSchmidtOrthogonalization()
[0] 2 16032 KSPSetUp_FGMRES()
[0] 4 16084160 KSPSetUp_GMRES()
[0] 2 36864 MatColoringApply_SL()
[0] 1 656 MatColoringCreate()
[0] 6 17088 MatCreate()
[0] 1 16 MatCreateMFFD_WP()
[0] 1 16 MatCreateSubMatrices_SeqBAIJ()
[0] 1 12288 MatCreateSubMatrix_SeqBAIJ()
[0] 3 32320 MatCreateSubMatrix_SeqBAIJ_Private()
[0] 2 1472 MatCreate_MFFD()
[0] 1 416 MatCreate_SeqAIJ()
[0] 3 864 MatCreate_SeqBAIJ()
[0] 2 416 MatCreate_Shell()
[0] 1 784 MatFDColoringCreate()
[0] 2 12288 MatFDColoringDegreeSequence_Minpack()
[0] 6 30859392 MatFDColoringSetUp_SeqXAIJ()
[0] 3 42512 MatGetColumnIJ_SeqAIJ()
[0] 4 72720 MatGetColumnIJ_SeqBAIJ_Color()
[0] 1 6144 MatGetOrdering_Natural()
[0] 2 36384 MatGetRowIJ_SeqAIJ()
[0] 7 210626000 MatILUFactorSymbolic_SeqBAIJ()
[0] 2 313376 MatIncreaseOverlap_SeqBAIJ()
[0] 2 30740608 MatLUFactorNumeric_SeqBAIJ_N()
[0] 1 6144 MatMarkDiagonal_SeqAIJ()
[0] 1 6144 MatMarkDiagonal_SeqBAIJ()
[0] 8 256 MatRegisterRootName()
[0] 1 6160 MatSeqAIJCheckInode()
[0] 4 115216 MatSeqAIJSetPreallocation_SeqAIJ()
[0] 4 302779424 MatSeqBAIJSetPreallocation_SeqBAIJ()
[0] 13 576 MatSolverTypeRegister()
[0] 1 16 PCASMCreateSubdomains()
[0] 2 1664 PCCreate()
[0] 1 160 PCCreate_ASM()
[0] 1 192 PCCreate_ILU()
[0] 5 307264 PCSetUp_ASM()
[0] 2 416 PetscBTCreate()
[0] 2 3216 PetscClassPerfLogCreate()
[0] 2 1616 PetscClassRegLogCreate()
[0] 2 32 PetscCommBuildTwoSided_Allreduce()
[0] 2 64 PetscCommDuplicate()
[0] 2 1888 PetscDSCreate()
[0] 2 26416 PetscEventPerfLogCreate()
[0] 2 158400 PetscEventPerfLogEnsureSize()
[0] 2 1616 PetscEventRegLogCreate()
[0] 2 9600 PetscEventRegLogRegister()
[0] 8 102400 PetscFreeSpaceGet()
[0] 474 15168 PetscFunctionListAdd_Private()
[0] 2 528 PetscIntStackCreate()
[0] 142 11360 PetscLayoutCreate()
[0] 56 896 PetscLayoutSetUp()
[0] 59 9440 PetscObjectComposedDataIncreaseReal()
[0] 2 576 PetscObjectListAdd()
[0] 33 768 PetscOptionsGetEList()
[0] 1 16 PetscOptionsHelpPrintedCreate()
[0] 1 32 PetscPushSignalHandler()
[0] 7 6944 PetscSFCreate()
[0] 3 432 PetscSFCreate_Basic()
[0] 2 1472 PetscSFLinkCreate()
[0] 11 1229040 PetscSFSetUpRanks()
[0] 7 614512 PetscSFSetUp_Basic()
[0] 4 20096 PetscSegBufferCreate()
[0] 2 1488 PetscSplitReductionCreate()
[0] 2 3008 PetscStageLogCreate()
[0] 1148 23872 PetscStrallocpy()
[0] 6 13056 PetscStrreplace()
[0] 9 3456 PetscTableCreate()
[0] 1 16 PetscViewerASCIIOpen()
[0] 6 96 PetscViewerAndFormatCreate()
[0] 1 752 PetscViewerCreate()
[0] 1 96 PetscViewerCreate_ASCII()
[0] 2 1424 SNESCreate()
[0] 1 16 SNESCreate_NEWTONLS()
[0] 1 1008 SNESLineSearchCreate()
[0] 1 16 SNESLineSearchCreate_BT()
[0] 16 1824 SNESMSRegister()
[0] 46 9056 TSARKIMEXRegister()
[0] 1 1264 TSAdaptCreate()
[0] 8 384 TSBasicSymplecticRegister()
[0] 1 2160 TSCreate()
[0] 1 224 TSCreate_Theta()
[0] 48 5968 TSGLEERegister()
[0] 41 7728 TSRKRegister()
[0] 89 14736 TSRosWRegister()
[0] 71 110192 VecCreate()
[0] 1 307200 VecCreateGhostWithArray()
[0] 123 36874080 VecCreate_MPI_Private()
[0] 7 4300800 VecCreate_Seq()
[0] 8 256 VecCreate_Seq_Private()
[0] 6 400 VecDuplicateVecs_Default()
[0] 3 2352 VecScatterCreate()
[0] 7 1843296 VecScatterSetUp_SF()
[0] 126 2016 VecStashCreate_Private()
[0] 1 3072 mapBlockColoringToJacobian()
On Wed, Aug 12, 2020 at 4:22 PM Barry Smith <bsmith at petsc.dev> wrote:
>
> Yes, there are some PETSc objects or arrays that you are not freeing so
> they are printed at the end of the run. For small runs this harmless but if
> new objects/memory is allocated at each iteration and not suitably freed it
> will eventually add up.
>
> Run with -malloc_view (small problem with say 2 iterations) it will
> print everything allocated and might be helpful.
>
> Perhaps you are calling ISColoringGetIS() and not calling
> ISColoringRestoreIS()?
>
> It is also possible it is a leak in PETSc, but that is unlikely since
> we test for them.
>
> Are you using Fortran?
>
> Barry
>
>
> On Aug 12, 2020, at 1:29 PM, Mark Lohry <mlohry at gmail.com> wrote:
>
> Thanks Matt and Barry. At Matt's suggestion I ran a smaller representative
> case with valgrind and didn't see anything alarming (apart from a small
> leak in an older boost version I was using:
> https://github.com/boostorg/serialization/issues/104 although I don't
> think this was causing the issue).
>
> -malloc_debug dumps quite a lot, this is supposed to be empty right?
> Output pasted below. It looks like the same sequence of calls is repeated 8
> times, which is how many nonlinear solves occurred in this particular run.
> Thoughts?
>
>
>
> [ 0]1408 bytes PetscSplitReductionCreate() line 63 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c
> [ 0]80 bytes PetscSplitReductionCreate() line 57 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/vec/utils/comb.c
> [ 0]16 bytes PetscCommBuildTwoSided_Allreduce() line 169 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/mpits.c
> [ 0]16 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]272 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]880 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]960 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]976 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]1024 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]1040 bytes ISGeneralSetIndices_General() line 578 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]16 bytes PetscLayoutSetUp() line 269 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]80 bytes PetscLayoutCreate() line 55 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/utils/pmap.c
> [ 0]16 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 255 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]32 bytes PetscStrallocpy() line 187 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/utils/str.c
> [ 0]32 bytes PetscFunctionListAdd_Private() line 222 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/dll/reg.c
> [ 0]16 bytes ISCreate_General() line 647 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/impls/general/general.c
> [ 0]896 bytes ISCreate() line 37 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/interface/isreg.c
> [ 0]64 bytes ISColoringGetIS() line 266 in
> /home/mlohry/dev/cmake-build/external/petsc/src/vec/is/is/utils/iscoloring.c
> [ 0]32 bytes PetscCommDuplicate() line 129 in
> /home/mlohry/dev/cmake-build/external/petsc/src/sys/objects/tagm.c
>
>
>
> On Wed, Aug 12, 2020 at 1:46 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>> Mark.
>>
>> When valgrind is not feasible (like on many centrally controlled
>> batch systems) you can run PETSc with an extra flag to do some memory error
>> checks
>> -malloc_debug
>>
>> this
>>
>> 1) fills all malloced memory with Nan so if the code is using
>> uninitialized memory it may be detected and
>> 2) checks the beginning and end of each alloced memory region for
>> out-of-bounds writes at each malloc and free.
>>
>> it will slow the code down a little bit but generally not a huge amount.
>>
>> It is no where near as good as valgrind or other memory corruption tools
>> but it has the advantage you can run it anywhere on any size job.
>>
>>
>> Barry
>>
>>
>>
>>
>>
>> On Aug 12, 2020, at 7:46 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Wed, Aug 12, 2020 at 7:53 AM Mark Lohry <mlohry at gmail.com> wrote:
>>
>>> I'm getting seemingly random failures of late:
>>> Caught signal number 7 BUS: Bus Error, possibly illegal memory access
>>>
>>
>> The first thing I would do is run valgrind on as wide an array of tests
>> as you can. This will find problems
>> on things that run completely fine.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> Symptoms:
>>> 1) Seems to only happen (so far) on larger cases, 400-2000 cores
>>> 2) It doesn't happen right away -- this was running happily for several
>>> hours over several hundred time steps with no indication of bad health in
>>> the numerics
>>> 3) At least the total memory consumption seems to be within bounds,
>>> though I'm not sure about individual processes. e.g. slurm here reported
>>> Memory Efficiency: 75.23% of 1.76 TB (180.00 GB/node)
>>> 4) running the same setup twice it fails at different points
>>>
>>> Any suggestions on what to look for? This is a bit painful to work on as
>>> I can only reproduce it on large runs and then it's seemingly random.
>>>
>>>
>>> Thanks,
>>> Mark
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200812/ffebe957/attachment-0001.html>
More information about the petsc-users
mailing list