<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 26, 2020, at 5:58 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 26, 2020 at 5:44 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Sep 26, 2020 at 1:07 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Sat, Sep 26, 2020 at 11:17 AM Mark McClure <<a href="mailto:mark@resfrac.com" target="_blank" class="">mark@resfrac.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Thank you, all for the explanations. <div class=""><br class=""></div><div class="">Following Matt's suggestion, we'll use -g (and not use -with-debugging=0) all future compiles to all users, so in future, we can provide better information.</div><div class=""><br class=""></div><div class="">Second, Chris is going to boil our function down to minimum stub and share in case there is some subtle issue with the way the functions are being called. </div><div class=""><br class=""></div><div class="">Third, I have question/request - Petsc is, in fact, detecting an error. As far as I can tell, this is not an uncontrolled 'seg fault'. It seems to me that maybe Petsc could choose to return out from the function when it detects this error, returning an error code, rather than dumping the core and terminating the program. If Petsc simply returned out with an error message, this would resolve the problem for us. After the Petsc call, we check for Petsc error messages. If Petsc returns an error - that's fine - we use a direct solver as a backup, and the simulation continues. So - I am not sure whether this is feasible - but if Petsc could return out with an error message - rather than dumping the core and terminating the program - then that would effectively resolve the issue for us. Would this change be possible?</div></div></blockquote><div class=""><br class=""></div><div class="">At some level, I think it is currently doing what you want. CHKERRQ() simply returns an error code from that function call, printing an error message. Suppressing the message is harder I think,</div></div></div></blockquote><div class=""><br class=""></div><div class="">He does not need this.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class="">but for now, if you know what function call is causing the error, you can just catch the (ierr != 0) yourself instead of using CHKERRQ. </div></div></div></blockquote><div class=""><br class=""></div><div class="">This is what I suggested earlier but maybe I was not clear enough.</div><div class=""><br class=""></div><div class="">Your code calls something like </div><div class=""><br class=""></div><div class="">ierr = SNESSolve(....); CHKERRQ(ierr);</div><div class=""><br class=""></div><div class="">You can replace this with:</div><div class=""><br class=""></div><div class=""> ierr = SNESSolve(....); <br class=""></div><div class=""> if (ierr) {</div></div></div></blockquote><div class="">How to deal with CHKERRQ(ierr); inside SNESSolve()?</div></div></div></div></blockquote><div><br class=""></div><div><br class=""></div> PetscPushErrorHandler(PetscIgnoreErrorHandler,NULL);</div><div><br class=""></div><div> But the problem in this code's runs appear to be due to corrupt data, why and how it gets corrupted is not known. Continuing with an alternative solver because a solver failed for numerical or algorithmic reasons is generally fine but continuing when there is corrupted data is always iffy because one doesn't know how far the corruption has spread. SNESDestroy(&snes); SNESCreate(&snes); may likely clean out any potentially corrupted data but if the corruption got into the mesh data structures it will still be there.</div><div><br class=""></div><div> A very sophisticated library code would, when it detects this type of corruption, sweep through all the data structures looking for any indications of corruption, to help determine the cause and/or even fix the problem. We don't have this code in place, though we could add some, because generally we relay on valgrind or -malloc_debug to detect such corruption, the problem is valgrind and -malloc_debug don't fit well in a production environment. Handling corruption that comes up in production but not testing is difficult.</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class=""> .... </div><div class=""> }</div><div class=""><br class=""></div><div class="">I suggested something earlier to do here. Maybe call KSPView. You could even destroy the solver and start the solver from scratch and see if that works.</div><div class=""><br class=""></div><div class="">Mark</div><div class=""> <br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="gmail_quote"><div class="">The drawback here is that we might not have cleaned up</div><div class="">all the state so that restarting makes sense. It should be possible to just kill the solve, reset the solver, and retry, although it is not clear to me at first glance if MPI will be in an okay state.</div><div class=""><br class=""></div></div></div>
</blockquote></div></div>
</blockquote></div></div>
</div></blockquote></div><br class=""></body></html>