<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 29, 2020, at 1:07 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Ah yes the 10-100 merged.<div class="">And I am not calling the <b class="">GPU timers </b>so the Mflops is messed up.</div></div></div></blockquote><div><br class=""></div> <blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class=""><div class="">And, I assume WaitForCUDA blocks on this MPI process's Cuda calls here. One stream per process. This does not block with other MPI process Cuda calls.</div></div></div></div></blockquote><div><br class=""></div> Not sure what you mean by this. The WaitForCUDA() waits on the CPU until the device has finished the kernel (not related to other MPI processes). It is an attempt to get a reasonably accurate timing for JUST the GPU kernel run and not include other things related to the CPU. (of course it does include the extra sync time).</div><div><br class=""></div><div> Hong Zhang is working on a better timer that uses CUDA event timers that will be more accurate and do not have the extra overhead of the device synch. Once that is merged it will be the default and the WaitForCUDA() will no longer be needed or used.</div><div><br class=""></div><div> Barry</div><div><br class=""></div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><br class=""></div><div class=""></div></div><div class=""> err = WaitForCUDA();CHKERRCUDA(err);<br class=""><b class=""> ierr = PetscLogGpuTimeEnd();CHKERRQ(ierr);<br class=""></b> ierr = PetscLogEventEnd(MAT_CUSPARSEGenerateTranspose,A,0,0,0);CHKERRQ(ierr);<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thanks,</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 29, 2020 at 1:12 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""></div> Mark,<div class=""><br class=""></div><div class=""> Aside from formatting are you sure there is an issue. Could it be 100 % of the time and 100% of the flops and since there is not room for all the digits they end up sitting on top of each other? </div><div class=""><br class=""></div><div class=""> Similarly could the flop rates be overlapped on top of each other? You could try adding more digits in the print statement to make room for these values.</div><div class=""><b style="font-family:monospace" class=""><br class=""></b></div><div class=""><b style="font-family:monospace" class=""> Barry</b></div><div class=""><b style="font-family:monospace" class=""> </b><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Dec 29, 2020, at 8:32 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">I am seeing this from a GPU kernel. The % flops is messed up and the flop rate does not look right:<div class=""><br class=""><div class=""><font face="monospace" class="">------------------------------------------------------------------------------------------------------------------------<br class="">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU<br class=""> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F<br class="">---------------------------------------------------------------------------------------------------------------------------------------------------------------<br class=""></font></div><div class=""><font face="monospace" class="">Jac-kernel 13068 1.0 1.6106e+01 1.1 6.13e+13 1.0 0.0e+00 0.0e+00 0.0e+00 <b class="">10100</b> 0 0 0 <b class="">10100</b> 0 0 0 <b class="">136983459</b> 0 0 0.00e+00 0 0.00e+00 100</font><br class=""></div></div><div class=""><font face="monospace" class=""><br class=""></font></div><div class=""><font face="monospace" class="">I use this in <a href="http://landau.cu/" target="_blank" class="">landau.cu</a>:</font></div><div class=""><font face="monospace" class=""><br class=""></font></div><div class=""> ierr = PetscLogGpuFlops(flops*nip);CHKERRQ(ierr);<font face="monospace" class=""><br class=""></font></div><div class=""><br class=""></div><div class="">Any idea what is going on here?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark</div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
</div></blockquote></div><br class=""></body></html>