<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Only KSPSolve, SNESSolve, TSSolve will have legitimate values. "High-level" functions like PtAP can be asynchronous (meaning the GPU returns to the CPU to do more stuff) before they are complete, hence their timings would be incorrect and so they must be labeled with a special marker and not have numbers for time.<div class=""><br class=""></div><div class=""> Jed reports a possible 10% smaller time for KSPSolve() in this mode in some of his runs, compared to timing correctly the inner operations, and thus feels it is the best default.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 27, 2022, at 10:08 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 26, 2022 at 8:00 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> The current nan output has to be replaced to get the column alignment correct, I just didn't feel like making that change also in the same MR. <div class=""><br class=""></div><div class=""> Something like Unknown or anything that fits in the column space would be fine. It just means for the given run the timing numbers are not meaningful/correct for those events. </div></div></blockquote><div class=""><br class=""></div><div class="">Just a note, just about every event is NAN for me. My GAMG setup that is all CPU is NAN. High level functions like PtAP as well. </div><div class="">That said, adding <span style="color:rgb(85,85,85);letter-spacing:0.2px;white-space:nowrap" class="">-log_view_gpu_time is fine. Not worth the churn.</span></div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">This is to obtain the best meaningful results for the outer events per Jed since timing the inner events accurately introduces extra time in the outer events. That is it is not possible to have the best accurate times for both inner events and outer events in the same run. So if you want to compare KSPSolve timings, for example, you run as-is, it you want to examine, low-level vector operations run also with -log_view_gpu_time but know that the KSP times are higher than need be.</div><div class=""><br class=""></div><div class=""> Sorry for the confusion.</div><div class=""><br class=""></div><div class=""><br class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Apr 26, 2022, at 3:49 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Tue, Apr 26, 2022 at 12:03 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Well, Nans are a clear sign that something is very wrong.</div></blockquote><div class=""><br class=""></div><div class="">Barry chose them so that it could not be mistaken for an actual number.</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 26, 2022 at 11:52 AM Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" target="_blank" class="">jacob.fai@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">There is an automatic warning that shows when you do run with `-log_view_gpu_time`, but perhaps there should also be an automatic warning when *not* running with it. It is unfortunate that NaN is the value printed as this implies a bug but AFAIK it is unavoidable (Barry can say more on this though).<br class="">
<br class="">
Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
<br class="">
> On Apr 26, 2022, at 09:48, Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>> wrote:<br class="">
> <br class="">
> You have to add -log_view_gpu_time<br class="">
> See <a href="https://gitlab.com/petsc/petsc/-/merge_requests/5056" rel="noreferrer" target="_blank" class="">https://gitlab.com/petsc/petsc/-/merge_requests/5056</a><br class="">
> <br class="">
> Jose<br class="">
> <br class="">
> <br class="">
>> El 26 abr 2022, a las 16:39, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> escribió:<br class="">
>> <br class="">
>> I'm seeing this on Perlmutter with Kokkos-CUDA. Nans in most log timing data except the two 'Solve' lines.<br class="">
>> Just cg/jacobi on snes/ex56.<br class="">
>> <br class="">
>> Any ideas?<br class="">
>> <br class="">
>> VecTDot 2 1.0 nan nan 1.20e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecNorm 2 1.0 nan nan 1.00e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecCopy 2 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> VecSet 5 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> VecAXPY 4 1.0 nan nan 2.40e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> VecPointwiseMult 1 1.0 nan nan 3.00e+00 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> KSPSetUp 1 1.0 nan nan 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 -nan -nan 0 0.00e+00 0 0.00e+00 0<br class="">
>> KSPSolve 1 1.0 4.0514e-04 1.0 5.50e+01 1.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 2 0 0 0 0 0 -nan 0 0.00e+00 0 0.00e+00 100<br class="">
>> SNESSolve 1 1.0 2.2128e-02 1.0 5.55e+05 1.0 0.0e+00 0.0e+00 0.0e+00 72 56 0 0 0 100100 0 0 0 25 -nan 0 0.00e+00 0 0.00e+00 0<br class="">
> <br class="">
<br class="">
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div></div>
</div></blockquote></div><br class=""></div></body></html>