<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Mark.<div class=""><br class=""></div><div class=""> When valgrind is not feasible (like on many centrally controlled batch systems) you can run PETSc with an extra flag to do some memory error checks</div><div class=""> -malloc_debug</div><div class=""><br class=""></div><div class=""> this </div><div class=""><br class=""></div><div class="">1) fills all malloced memory with Nan so if the code is using uninitialized memory it may be detected and </div><div class="">2) checks the beginning and end of each alloced memory region for out-of-bounds writes at each malloc and free.</div><div class=""><br class=""></div><div class="">it will slow the code down a little bit but generally not a huge amount.</div><div class=""><br class=""></div><div class="">It is no where near as good as valgrind or other memory corruption tools but it has the advantage you can run it anywhere on any size job.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 12, 2020, at 7:46 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Wed, Aug 12, 2020 at 7:53 AM Mark Lohry <<a href="mailto:mlohry@gmail.com" class="">mlohry@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">I'm getting seemingly random failures of late:</div><div class="">Caught signal number 7 BUS: Bus Error, possibly illegal memory access</div></div></blockquote><div class=""><br class=""></div><div class="">The first thing I would do is run valgrind on as wide an array of tests as you can. This will find problems</div><div class="">on things that run completely fine.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Symptoms:</div><div class="">1) Seems to only happen (so far) on larger cases, 400-2000 cores</div><div class="">2) It doesn't happen right away -- this was running happily for several hours over several hundred time steps with no indication of bad health in the numerics</div><div class="">3) At least the total memory consumption seems to be within bounds, though I'm not sure about individual processes. e.g. slurm here reported Memory Efficiency: 75.23% of 1.76 TB (180.00 GB/node)</div><div class="">4) running the same setup twice it fails at different points<br class=""></div><div class=""><br class=""></div><div class="">Any suggestions on what to look for? This is a bit painful to work on as I can only reproduce it on large runs and then it's seemingly random.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark<br class=""></div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></body></html>