<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 26, 2022, at 6:40 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Tue, Apr 26, 2022 at 6:12 PM Justin Chang <<a href="mailto:jychang48@gmail.com" class="">jychang48@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I think N/A (not applicable) would be a better message than NaN. Prior to this mail, even I thought I broke something with these NaN's</div></blockquote><div class=""><br class=""></div><div class="">We would need some logic to check for NaN and change the format.</div></div></div></div></blockquote><div><br class=""></div> Yes, that should to in another MR.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 26, 2022 at 2:49 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">On Tue, Apr 26, 2022 at 12:03 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Well, Nans are a clear sign that something is very wrong.</div></blockquote><div class=""><br class=""></div><div class="">Barry chose them so that it could not be mistaken for an actual number.</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 26, 2022 at 11:52 AM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">There is an automatic warning that shows when you do run with `-log_view_gpu_time`, but perhaps there should also be an automatic warning when *not* running with it. It is unfortunate that NaN is the value printed as this implies a bug but AFAIK it is unavoidable (Barry can say more on this though).<br class="">
<br class="">
Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
<br class="">
> On Apr 26, 2022, at 09:48, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> wrote:<br class="">
> <br class="">
> You have to add -log_view_gpu_time<br class="">
> See <a href="https://gitlab.com/petsc/petsc/-/merge_requests/5056" rel="noreferrer" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/5056</a><br class="">
> <br class="">
> Jose<br class="">
> <br class="">
> <br class="">
>> El 26 abr 2022, a las 16:39, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> escribió:<br class="">
>> <br class="">
>> I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing data except the two 'Solve' lines.<br class="">
>> Just cg/jacobi on snes/ex56.<br class="">
>> <br class="">
>> Any ideas?<br class="">
>> <br class="">
>> VecTDot 2 1.0 nan nan 1.20e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecNorm 2 1.0 nan nan 1.00e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecCopy 2 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> VecSet 5 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> VecAXPY 4 1.0 nan nan 2.40e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecPointwiseMult 1 1.0 nan nan 3.00e+00 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> KSPSetUp 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> KSPSolve 1 1.0 4.0514e-04 1.0 5.50e+01 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> SNESSolve 1 1.0 2.2128e-02 1.0 5.55e+05 1.0 0.0e+00 0.0e+00 0.0e+00 72 56 0 0 0 100100 0 0 0 25 -nan 0 0.00e+00 0 0.00e+00 0<br class="">
> <br class="">
<br class="">
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></body></html>