[petsc-users] Trying to understand -log_view when using HIP kernels (ex34)

[Junchao Zhang]

Try to also add -log_view_gpu_time,
https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/

--Junchao Zhang


On Fri, Jan 19, 2024 at 11:35 AM Dave May <dave.mayhem23 at gmail.com> wrote:

> Hi all,
>
> I am trying to understand the logging information associated with the
> %flops-performed-on-the-gpu reported by -log_view when running
>   src/ksp/ksp/tutorials/ex34
> with the following options
> -da_grid_x 192
> -da_grid_y 192
> -da_grid_z 192
> -dm_mat_type seqaijhipsparse
> -dm_vec_type seqhip
> -ksp_max_it 10
> -ksp_monitor
> -ksp_type richardson
> -ksp_view
> -log_view
> -mg_coarse_ksp_max_it 2
> -mg_coarse_ksp_type richardson
> -mg_coarse_pc_type none
> -mg_levels_ksp_type richardson
> -mg_levels_pc_type none
> -options_left
> -pc_mg_levels 3
> -pc_mg_log
> -pc_type mg
>
> This config is not intended to actually solve the problem, rather it is a
> stripped down set of options designed to understand what parts of the
> smoothers are being executed on the GPU.
>
> With respect to the log file attached, my first set of questions related
> to the data reported under "Event Stage 2: MG Apply".
>
> [1] Why is the log littered with nan's?
> * I don't understand how and why "GPU Mflop/s" should be reported as nan
> when a value is given for "GPU %F" (see MatMult for example).
>
> * For events executed on the GPU, I assume the column "Time (sec)" relates
> to "CPU execute time", this would explain why we see a nan in "Time (sec)"
> for MatMult.
> If my assumption is correct, how should I interpret the column "Flop
> (Max)" which is showing 1.92e+09?
> I would assume of "Time (sec)" relates to the CPU then "Flop (Max)" should
> also relate to CPU and GPU flops would be logged in "GPU Mflop/s"
>
> [2] More curious is that within "Event Stage 2: MG Apply" KSPSolve,
> MGSmooth Level 0, MGSmooth Level 1, MGSmooth Level 2 all report "GPU %F" as
> 93. I believe this value should be 100 as the smoother (and coarse grid
> solver) are configured as richardson(2)+none and thus should run entirely
> on the GPU.
> Furthermore, when one inspects all events listed under "Event Stage 2: MG
> Apply" those events which do flops correctly report "GPU %F" as 100.
> And the events showing "GPU %F" = 0 such as
>   MatHIPSPARSCopyTo, VecCopy, VecSet, PCApply, DCtxSync
> don't do any flops (on the CPU or GPU) - which is also correct
> (although non GPU events should show nan??)
>
> Hence I am wondering what is the explanation for the missing 7% from "GPU
> %F" for KSPSolve and MGSmooth {0,1,2}??
>
> Does anyone understand this -log_view, or can explain to me how to
> interpret it?
>
> It could simply be that:
> a) something is messed up with -pc_mg_log
> b) something is messed up with the PETSc build
> c) I am putting too much faith in -log_view and should profile the code
> differently.
>
> Either way I'd really like to understand what is going on.
>
>
> Cheers,
> Dave
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20240119/98f8f0c3/attachment.html>