[petsc-users] PetscLogView produces nan's instead of timing data when using GPUs

[Sajid Ali]

Hi Barry,

Thanks for the explanation.

On Wed, Oct 5, 2022 at 4:11 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>   It prints Nan to indicate that the time for that event is not known
> accurately. But the times for the larger events that contain these events
> are known. So for example the time for KSPSolve is know but not the time
> for VecNorm.  The other numbers in the events, like number of times called
> etc that are not Nan are correct as displayed.
>
>   This is done because correctly tracking the times of the individual
> events requires synchronizations that slow down the entire calculation a
> bit; for example the time for the KSPSolve will register a longer time then
> it registers if the smaller events are not timed.
>
>   To display the times of the smaller events use -log_view_gpu_time also
> but note this will increase the times of the larger events a bit.
>
>   Barry
>
>
> On Oct 5, 2022, at 4:47 PM, Sajid Ali <sajidsyed2021 at u.northwestern.edu>
> wrote:
>
> Hi PETSc-developers,
>
> I'm having trouble with getting performance logs from an application that
> uses PETSc. There are no issues when I run it on a CPU, but every time a
> GPU is used there is no timing data and almost all times are replaced by
> times that are just `nan` (on two different clusters). I am attaching the
> log files for both cases with this email. Could someone explain what is
> happening here ?
>
> In case it helps, here are the routines used to initialize/finalize the
> application that also handle initializing/finalizing PETSc and printing the
> PETSc performance logs to PETSC_VIEWER_STDOUT_WORLD :
> https://github.com/fnalacceleratormodeling/synergia2/blob/devel3/src/synergia/utils/utils.h
>
> Thank You,
> Sajid Ali (he/him) | Research Associate
> Scientific Computing Division
> Fermi National Accelerator Laboratory
> s-sajid-ali.github.io
> <log-gpu><log-cpu>
>
>
>

-- 
Thank You,
Sajid Ali (he/him) | Research Associate
Scientific Computing Division
Fermi National Accelerator Laboratory
s-sajid-ali.github.io
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20221006/c8bb5d0f/attachment.html>