<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Also, maybe run with -info dump and grep for MUMPS errors in dump.%p, because some failures are silent otherwise.<div class=""><br class=""></div><div class="">Thanks,</div><div class="">Pierre<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 22 Mar 2021, at 7:09 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="">On Mon, Mar 22, 2021 at 2:07 PM Chris Hewson <<a href="mailto:chris@resfrac.com" class="">chris@resfrac.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class="">Hi Matt,<div class=""><br class=""></div><div class="">No, we are running it without debugging in prod and then running debug I can't reproduce the error, from stderr we get:</div><div class=""><br class=""></div>[1]PETSC ERROR: ------------------------------------------------------------------------<br class="">[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br class="">[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br class="">[1]PETSC ERROR: or see<span class="Apple-converted-space"> </span><a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br class="">[1]PETSC ERROR: or try<span class="Apple-converted-space"> </span><a href="http://valgrind.org/" target="_blank" class="">http://valgrind.org</a><span class="Apple-converted-space"> </span>on GNU/linux and Apple Mac OS X to find memory corruption errors<br class="">[1]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<span class="Apple-converted-space"> </span><br class="">[1]PETSC ERROR: to get more information on the crash.<br class="">[1]PETSC ERROR: Run with -malloc_debug to check if memory corruption is causing the crash.<br class=""><div class="">application called MPI_Abort(MPI_COMM_WORLD, 50176059) - process 1 </div></div></blockquote><div class=""><br class=""></div><div class="">If you can afford it, running an instance with -on_error_attach_debugger so that if it fails we can get a stack trace, would be</div><div class="">very valuable, since right now we do not know exactly what is failing.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 22, 2021 at 12:04 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div dir="ltr" class="">On Mon, Mar 22, 2021 at 1:56 PM Chris Hewson <<a href="mailto:chris@resfrac.com" target="_blank" class="">chris@resfrac.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class="">Hi All,<div class=""><br class=""></div><div class="">I have been having a problem with MUMPS randomly crashing in our program and causing the entire program to crash. I am compiling in -O2 optimization mode and using --download-mumps etc. to compile PETSc. If I rerun the program, 95%+ of the time I can't reproduce the error. It seems to be a similar issue to this thread:</div><div class=""><br class=""></div><div class=""><a href="https://lists.mcs.anl.gov/pipermail/petsc-users/2018-October/036372.html" target="_blank" class="">https://lists.mcs.anl.gov/pipermail/petsc-users/2018-October/036372.html</a></div><div class=""><br class=""></div><div class="">Similar to the resolution there I am going to try and increase icntl_14 and see if that resolves the issue. Any other thoughts on this?</div></div></blockquote><div class=""><br class=""></div><div class="">When it fails, do you get a stack trace?</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr" class=""><div class="">Thanks,</div><div class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><b class=""><br class=""></b></div><div dir="ltr" class=""><b class="">Chris Hewson</b><div class="">Senior Reservoir Simulation Engineer</div><div class="">ResFrac</div><div class="">+1.587.575.9792</div></div></div></div></div></div></div></div></div></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div></blockquote></div></blockquote></div><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div><br class=""></div></body></html>