<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Junchao,<div><br></div><div> How come vecseqcupm_impl.hpp has PetscCall(PetscLogFlops(n)); instead of logging the flops on the GPU? </div><div><br></div><div> This could be the root of the problem, the VecShift used to remove the null space from vectors in the solver is logging incorrectly. (For some reason there is no LogEventBegin/End() for VecShift which is why it doesn't get it on line in the -log_view).</div><div><br></div><div> Barry</div><div><br></div><div><br></div><div><br></div><div><div><br><blockquote type="cite"><div>On Jan 19, 2024, at 3:17 PM, Barry Smith <bsmith@petsc.dev> wrote:</div><br class="Apple-interchange-newline"><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Junchao<div><br></div><div> I run the following on the CI machine, why does this happen? With trivial solver options it runs ok.</div><div><br></div><div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">bsmith@petsc-gpu-02:/scratch/bsmith/petsc/src/ksp/ksp/tutorials$ ./ex34 -da_grid_x 192 -da_grid_y 192 -da_grid_z 192 -dm_mat_type seqaijhipsparse -dm_vec_type seqhip -ksp_max_it 10 -ksp_monitor -ksp_type richardson -ksp_view -log_view -mg_coarse_ksp_max_it 2 -mg_coarse_ksp_type richardson -mg_coarse_pc_type none -mg_levels_ksp_type richardson -mg_levels_pc_type none -options_left -pc_mg_levels 3 -pc_mg_log -pc_type mg</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(180, 36, 25);"><span style="font-variant-ligatures: no-common-ligatures"><b>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</b></span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: GPU error</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: hipSPARSE errorcode 3 (HIPSPARSE_STATUS_INVALID_VALUE)</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: Option left: name:-options_left (no value) source: command line</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: Petsc Release Version 3.20.3, unknown </span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: ./ex34 on a named petsc-gpu-02 by bsmith Fri Jan 19 14:15:20 2024</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: Configure options --package-prefix-hash=/home/bsmith/petsc-hash-pkgs --with-make-np=24 --with-make-test-np=8 --with-hipc=/opt/rocm-5.4.3/bin/hipcc --with-hip-dir=/opt/rocm-5.4.3 COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" HIPOPTFLAGS="-g -O" --with-cuda=0 --with-hip=1 --with-precision=double --with-clanguage=c --download-kokkos --download-kokkos-kernels --download-hypre --download-magma --with-magma-fortran-bindings=0 --download-mfem --download-metis --with-strict-petscerrorcode PETSC_ARCH=arch-ci-linux-hip-double</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #1 MatMultAddKernel_SeqAIJHIPSPARSE() at /scratch/bsmith/petsc/src/mat/impls/aij/seq/seqhipsparse/aijhipsparse.hip.cpp:3131</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #2 MatMultAdd_SeqAIJHIPSPARSE() at /scratch/bsmith/petsc/src/mat/impls/aij/seq/seqhipsparse/aijhipsparse.hip.cpp:3004</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #3 MatMultAdd() at /scratch/bsmith/petsc/src/mat/interface/matrix.c:2770</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #4 MatInterpolateAdd() at /scratch/bsmith/petsc/src/mat/interface/matrix.c:8603</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #5 PCMGMCycle_Private() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:87</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #6 PCMGMCycle_Private() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:83</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #7 PCApply_MG_Internal() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:611</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #8 PCApply_MG() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:633</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #9 PCApply() at /scratch/bsmith/petsc/src/ksp/pc/interface/precon.c:498</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #10 KSP_PCApply() at /scratch/bsmith/petsc/include/petsc/private/kspimpl.h:383</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #11 KSPSolve_Richardson() at /scratch/bsmith/petsc/src/ksp/ksp/impls/rich/rich.c:106</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #12 KSPSolve_Private() at /scratch/bsmith/petsc/src/ksp/ksp/interface/itfunc.c:906</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #13 KSPSolve() at /scratch/bsmith/petsc/src/ksp/ksp/interface/itfunc.c:1079</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: #14 main() at ex34.c:52</span></div><div style="margin: 0px; font-style: normal; font-variant-caps: normal; font-stretch: normal; font-size: 16px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;"><span style="font-variant-ligatures: no-common-ligatures">[0]PETSC ERROR: PETSc Option Table entries:</span></div><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures: no-common-ligatures"> Dave,</span></div><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures: no-common-ligatures"> Trying to debug the 7% now, but having trouble running, as you see above.</span></div><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures: no-common-ligatures"><br></span></div><div><br><blockquote type="cite"><div>On Jan 19, 2024, at 3:02 PM, Dave May <dave.mayhem23@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr">Thank you Barry and Junchao for these explanations. I'll turn on -log_view_gpu_time.<div><br></div><div>Do either of you have any thoughts regarding why the percentage of flop's being reported on the GPU is not 100% for MGSmooth Level {0,1,2} for this solver configuration?</div><div><br></div><div>This number should have nothing to do with timings as it reports the ratio of operations performed on the GPU and CPU, presumably obtained from PetscLogFlops() and PetscLogGpuFlops().</div><div><br></div><div>Cheers,</div><div>Dave</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 19 Jan 2024 at 11:39, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Try to also add -log_view_gpu_time, <a href="https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/" target="_blank">https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/</a><div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 19, 2024 at 11:35 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I am trying to understand the logging information associated with the %flops-performed-on-the-gpu reported by -log_view when running </div><div> src/ksp/ksp/tutorials/ex34<br></div><div>with the following options</div><div>-da_grid_x 192<br>-da_grid_y 192<br>-da_grid_z 192<br>-dm_mat_type seqaijhipsparse<br>-dm_vec_type seqhip<br>-ksp_max_it 10<br>-ksp_monitor<br>-ksp_type richardson<br>-ksp_view<br>-log_view<br>-mg_coarse_ksp_max_it 2<br>-mg_coarse_ksp_type richardson<br>-mg_coarse_pc_type none<br>-mg_levels_ksp_type richardson<br>-mg_levels_pc_type none<br>-options_left<br>-pc_mg_levels 3<br>-pc_mg_log<br>-pc_type mg<br></div><div><br></div><div>This config is not intended to actually solve the problem, rather it is a stripped down set of options designed to understand what parts of the smoothers are being executed on the GPU.</div><div><br></div><div>With respect to the log file attached, my first set of questions related to the data reported under "Event Stage 2: MG Apply".</div><div><br></div><div>[1] Why is the log littered with nan's?</div><div>* I don't understand how and why "GPU Mflop/s" should be reported as nan when a value is given for "GPU %F" (see MatMult for example).</div><div><br></div><div>* For events executed on the GPU, I assume the column "Time (sec)" relates to "CPU execute time", this would explain why we see a nan in "Time (sec)" for MatMult.</div><div>If my assumption is correct, how should I interpret the column "Flop (Max)" which is showing 1.92e+09? </div><div>I would assume of "Time (sec)" relates to the CPU then "Flop (Max)" should also relate to CPU and GPU flops would be logged in "GPU Mflop/s"<br></div><div><br></div><div>[2] More curious is that within "Event Stage 2: MG Apply" KSPSolve, MGSmooth Level 0, MGSmooth Level 1, MGSmooth Level 2 all report "GPU %F" as 93. I believe this value should be 100 as the smoother (and coarse grid solver) are configured as richardson(2)+none and thus should run entirely on the GPU. </div><div>Furthermore, when one inspects all events listed under "Event Stage 2: MG Apply" those events which do flops correctly report "GPU %F" as 100. </div><div>And the events showing "GPU %F" = 0 such as <br></div><div> MatHIPSPARSCopyTo, VecCopy, VecSet, PCApply, DCtxSync</div><div>don't do any flops (on the CPU or GPU) - which is also correct (although non GPU events should show nan??)<br></div><div><br></div><div>Hence I am wondering what is the explanation for the missing 7% from "GPU %F" for KSPSolve and MGSmooth {0,1,2}??</div><div><br></div><div>Does anyone understand this -log_view, or can explain to me how to interpret it?</div><div><br></div><div>It could simply be that:</div><div>a) something is messed up with -pc_mg_log</div><div>b) something is messed up with the PETSc build</div><div>c) I am putting too much faith in -log_view and should profile the code differently.</div><div><br></div><div>Either way I'd really like to understand what is going on.</div><div><br></div><div><br></div><div>Cheers,</div><div>Dave</div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br></div></body></html>