<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Hmm, it is crashing in PetscFunctionListDLAllPop_Private() which is very recent code with Jacob's name on it.<div><br></div><div> It is only run in debug mode, hence only crashes there. I don't know why it fails here, somehow memory was corrupted earlier? Try running with -malloc_debug or under valgrind?</div><div><br></div><div><div>static PetscErrorCode PetscFunctionListDLAllPop_Private(PetscFunctionList fl)</div><div>{</div><div> PetscFunctionBegin;</div><div> if (PetscDefined(USE_DEBUG) && !PetscDefined(HAVE_THREADSAFETY)) {</div><div> PetscFunctionListDLAll current = dlallhead, prev = NULL;</div><div><br></div><div> /* Remove this entry from the main DL list (if it is in it) */</div><div> while (current) {</div><div> const PetscFunctionListDLAll next = current->next;</div><div><br></div><div> if (current->data == fl) {</div><div> if (prev) {</div><div> // somewhere in the middle (or end) of the list</div><div> prev->next = next;</div><div> } else {</div><div> // prev = NULL implies current = dlallhead, so front of list</div><div> dlallhead = next;</div><div> }</div><div> PetscCall(PetscFree(current));</div><div> break;</div><div> }</div><div> prev = current;</div><div> current = next;</div><div> }</div><div> }</div><div> PetscFunctionReturn(PETSC_SUCCESS);</div><div>}</div></div><div><br></div><div><br><div><br><blockquote type="cite"><div>On Feb 9, 2023, at 1:56 PM, Sajid Ali Syed <sasyed@fnal.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; background-color: rgb(255, 255, 255);">Hi Barry,<span class="Apple-converted-space"> </span><br><br>The lack of line numbers is due to the fact that this build of PETSc was done via spack which installs it in a temporary directory before moving it to the final location.<span class="Apple-converted-space"> </span><br><br>I have removed that build and installed PETSc locally (albeit with a simpler configuration) and see the same bug. Logs for this configuration and the error trace with this build are attached with this email.<span class="Apple-converted-space"> </span><br></div><div class="elementToProof" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;"><br></div><div id="Signature"><div><div></div><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">Thank You,<br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size: 12.8px;">Sajid Ali (he/him) | Research Associate<br></div><div style="font-size: 12.8px;">Scientific Computing Division<br></div><div style="font-size: 12.8px;">Fermi National Accelerator Laboratory<br></div><span style="font-size: 12.8px;"><a href="http://s-sajid-ali.github.io/" target="_blank">s-sajid-ali.github.io</a></span></div></div></div></div></div></div><br></div></div></div></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1026.046875px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Thursday, February 9, 2023 12:02 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Sajid Ali Syed <<a href="mailto:sasyed@fnal.gov">sasyed@fnal.gov</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] KSP_Solve crashes in debug mode</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-break: after-white-space;"><div><br></div> Hmm, looks like your build may be funny? It is not in debug mode <div><br></div><div><div>frame #2: 0x000000010eda20c8 libpetsc.3.018.dylib`PetscHeaderDestroy_Private + 1436</div><div> frame #3: 0x000000010f10176c libpetsc.3.018.dylib`VecDestroy + 808</div><div> frame #4: 0x0000000110199f34 libpetsc.3.018.dylib`KSPSolve_Private + 512</div><div><br></div><div>In debugger mode it would show the line numbers where the crash occurred and help us determine the problem. I do note the -g being used by the compilers so cannot explain off hand why it does not display the debug information.</div><div><br></div><div> Barry</div><div><br></div><div><br><blockquote type="cite"><div>On Feb 9, 2023, at 12:42 PM, Sajid Ali Syed via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:</div><br class="x_Apple-interchange-newline"><div><div class="x_markdown-here-wrapper" data-md-url="outlook.office365.com" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><p style="margin: 0px 0px 1.2em !important;">Hi PETSc-developers,<span class="x_Apple-converted-space"> </span></p><p style="margin: 0px 0px 1.2em !important;">In our application we call KSP_Solve as part of a step to propagate a beam through a lattice. I am observing a crash within KSP_Solve for an application only after the 43rd call to KSP_Solve when building the application and PETSc in debug mode, full logs for which are attached with this email (1 MPI rank and 4 OMP threads were used, but this crash occurs with multiple MPI ranks as well ). I am also including the last few lines of the configuration for this build. This crash does not occur when building the application and PETSc in release mode.<span class="x_Apple-converted-space"> </span></p><p style="margin: 0px 0px 1.2em !important;">Could someone tell me what causes this crash and if anything can be done to prevent it? Thanks in advance.</p><p style="margin: 0px 0px 1.2em !important;">The configuration of this solver is here :<span class="x_Apple-converted-space"> </span><code style="font-size: 0.85em; font-family: Consolas, Inconsolata, Courier, monospace; margin: 0px 0.15em; padding: 0px 0.3em; white-space: pre-wrap; border: 1px solid rgb(234, 234, 234); background-color: rgb(248, 248, 248); border-radius: 3px; display: inline;">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fnalacceleratormodeling_synergia2_blob_sajid_features_openpmd-5Fbasic-5Fintegration_src_synergia_collective_space-5Fcharge-5F3d-5Ffd-5Futils.cc-23L273-2DL292&d=DwMFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=fZRIP-srEU-rOVw461XeAmMY4_3VyZ1mujQae_3pw33K9hMFg-5yyo_sXsQJ_1dn&s=YToKoRAfim51m6aN-z4FlzfM0UDy46D7QJh8B11shTg&e=">
https://github.com/fnalacceleratormodeling/synergia2/blob/sajid/features/openpmd_basic_integration/src/synergia/collective/space_charge_3d_fd_utils.cc#L273-L292</a></code></p><p style="margin: 0px 0px 1.2em !important;"><br>Thank You,<br>Sajid Ali (he/him) | Research Associate<br>Scientific Computing Division<br>Fermi National Accelerator Laboratory<br><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io_&d=DwMFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=fZRIP-srEU-rOVw461XeAmMY4_3VyZ1mujQae_3pw33K9hMFg-5yyo_sXsQJ_1dn&s=g4Ch_bfIjw1n_m0d0zSXYc_eDI-4eJc9zSA_RfhUFOQ&e=">s-sajid-ali.github.io</a></p><div title="MDH:PGRpdiBjbGFzcz0iZWxlbWVudFRvUHJvb2YiIHN0eWxlPSJmb250LWZhbWlseTpDYWxpYnJpLEFy
aWFsLEhlbHZldGljYSxzYW5zLXNlcmlmOyBmb250LXNpemU6MTJwdDsgY29sb3I6cmdiKDAsMCww
KTsgYmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1NSwyNTUpIj5IaSBQRVRTYy1kZXZlbG9wZXJz
LCA8YnI+PC9kaXY+PGRpdiBjbGFzcz0iZWxlbWVudFRvUHJvb2YgQ29udGVudFBhc3RlZDAiIHN0
eWxlPSJmb250LWZhbWlseTpDYWxpYnJpLEFyaWFsLEhlbHZldGljYSxzYW5zLXNlcmlmOyBmb250
LXNpemU6MTJwdDsgY29sb3I6cmdiKDAsMCwwKTsgYmFja2dyb3VuZC1jb2xvcjpyZ2IoMjU1LDI1
NSwyNTUpIj48YnI+SW4gb3VyIGFwcGxpY2F0aW9uIHdlIGNhbGwgS1NQX1NvbHZlIGFzIHBhcnQg
b2YgYSBzdGVwIHRvIHByb3BhZ2F0ZSBhIGJlYW0gdGhyb3VnaCBhIGxhdHRpY2UuIEkgYW0gb2Jz
ZXJ2aW5nIGEgY3Jhc2ggd2l0aGluIEtTUF9Tb2x2ZSBmb3IgYW4gYXBwbGljYXRpb24gb25seSBh
ZnRlciB0aGUgNDM8c3Bhbj48c3VwPnJkPC9zdXA+IGNhbGwgdG8gS1NQX1NvbHZlPC9zcGFuPiB3
aGVuIGJ1aWxkaW5nIHRoZSBhcHBsaWNhdGlvbiBhbmQgUEVUU2MgaW4gZGVidWcgbW9kZSwgZnVs
bCBsb2dzIGZvciB3aGljaCBhcmUgYXR0YWNoZWQgd2l0aCB0aGlzIGVtYWlsLiBJIGFtIGFsc28g
aW5jbHVkaW5nIHRoZSBsYXN0IGZldyBsaW5lcyBvZiB0aGUgY29uZmlndXJhdGlvbiBmb3IgdGhp
cyBidWlsZC4gVGhpcyBjcmFzaCBkb2VzIG5vdCBzZWVtIHRvIG9jY3VyIHdoZW4gYnVpbGRpbmcg
dGhlIGFwcGxpY2F0aW9uIGFuZCBQRVRTYyBpbiByZWxlYXNlIG1vZGUuIDxicj48YnI+Q291bGQg
c29tZW9uZSB0ZWxsIG1lIHdoYXQgY2F1c2VzIHRoaXMgY3Jhc2ggYW5kIGlmIGFueXRoaW5nIGNh
biBiZSBkb25lIHRvIHByZXZlbnQgaXQ/IFRoYW5rcyBpbiBhZHZhbmNlLjxicj48YnI+VGhlIGNv
bmZpZ3VyYXRpb24gb2YgdGhpcyBzb2x2ZXIgaXMgaGVyZSA6IGA8YSBocmVmPSJodHRwczovL2dp
dGh1Yi5jb20vZm5hbGFjY2VsZXJhdG9ybW9kZWxpbmcvc3luZXJnaWEyL2Jsb2Ivc2FqaWQvZmVh
dHVyZXMvb3BlbnBtZF9iYXNpY19pbnRlZ3JhdGlvbi9zcmMvc3luZXJnaWEvY29sbGVjdGl2ZS9z
cGFjZV9jaGFyZ2VfM2RfZmRfdXRpbHMuY2MjTDI3My1MMjkyIiBpZD0iTFBsbmsxMzIxNjMiPmh0
dHBzOi8vZ2l0aHViLmNvbS9mbmFsYWNjZWxlcmF0b3Jtb2RlbGluZy9zeW5lcmdpYTIvYmxvYi9z
YWppZC9mZWF0dXJlcy9vcGVucG1kX2Jhc2ljX2ludGVncmF0aW9uL3NyYy9zeW5lcmdpYS9jb2xs
ZWN0aXZlL3NwYWNlX2NoYXJnZV8zZF9mZF91dGlscy5jYyNMMjczLUwyOTI8L2E+YDxicj48L2Rp
dj48ZGl2IGNsYXNzPSJlbGVtZW50VG9Qcm9vZiI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6Q2Fs
aWJyaSxBcmlhbCxIZWx2ZXRpY2Esc2Fucy1zZXJpZjsgZm9udC1zaXplOjEycHQ7IGNvbG9yOnJn
YigwLDAsMCkiPjxicj48YnI+PC9kaXY+PGRpdiBpZD0iU2lnbmF0dXJlIj48ZGl2PjxkaXY+PC9k
aXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6Q2FsaWJyaSxBcmlhbCxIZWx2ZXRpY2Esc2Fucy1z
ZXJpZjsgZm9udC1zaXplOjEycHQ7IGNvbG9yOnJnYigwLDAsMCkiPlRoYW5rIFlvdSw8YnI+PGRp
diBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2IGRpcj0ibHRyIj48ZGl2PjxkaXYg
ZGlyPSJsdHIiPjxkaXYgc3R5bGU9ImZvbnQtc2l6ZToxMi44cHgiPlNhamlkIEFsaSAoaGUvaGlt
KSB8IFJlc2VhcmNoIEFzc29jaWF0ZTxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LXNpemU6MTIu
OHB4Ij5TY2llbnRpZmljIENvbXB1dGluZyBEaXZpc2lvbjxicj48L2Rpdj48ZGl2IHN0eWxlPSJm
b250LXNpemU6MTIuOHB4Ij5GZXJtaSBOYXRpb25hbCBBY2NlbGVyYXRvciBMYWJvcmF0b3J5PGJy
PjwvZGl2PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuOHB4Ij48YSBocmVmPSJodHRwOi8vcy1z
YWppZC1hbGkuZ2l0aHViLmlvIiBkYXRhLWF1dGg9Ik5vdEFwcGxpY2FibGUiPnMtc2FqaWQtYWxp
LmdpdGh1Yi5pbzwvYT48L3NwYW4+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+
PGJyPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pg==" style="height: 0px; width: 0px; max-height: 0px; max-width: 0px; overflow: hidden; font-size: 0em; padding: 0px; margin: 0px;"></div></div><span id="x_cid:AF637DA4-FA66-4915-80AE-6C9C68FFBA02"><ksp_crash_log></span></div></blockquote></div><br></div></div></div></blockquote></div></div></body></html>