<div dir="ltr">Thanks Barry!</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 19 Jan 2024 at 12:18, Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="line-break:after-white-space"><div><br></div>  Junchao<div><br></div><div>    I run the following on the CI machine, why does this happen? With trivial solver options it runs ok.</div><div><br></div><div><p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">bsmith@petsc-gpu-02:/scratch/bsmith/petsc/src/ksp/ksp/tutorials$ ./ex34 -da_grid_x 192 -da_grid_y 192 -da_grid_z 192 -dm_mat_type seqaijhipsparse -dm_vec_type seqhip -ksp_max_it 10 -ksp_monitor -ksp_type richardson -ksp_view -log_view -mg_coarse_ksp_max_it 2 -mg_coarse_ksp_type richardson -mg_coarse_pc_type none -mg_levels_ksp_type richardson -mg_levels_pc_type none -options_left -pc_mg_levels 3 -pc_mg_log -pc_type mg</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal;color:rgb(180,36,25)"><span style="font-variant-ligatures:no-common-ligatures"><b>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</b></span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: GPU error</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: hipSPARSE errorcode 3 (HIPSPARSE_STATUS_INVALID_VALUE)</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR:   Option left: name:-options_left (no value) source: command line</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Petsc Release Version 3.20.3, unknown </span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: ./ex34 on a  named petsc-gpu-02 by bsmith Fri Jan 19 14:15:20 2024</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Configure options --package-prefix-hash=/home/bsmith/petsc-hash-pkgs --with-make-np=24 --with-make-test-np=8 --with-hipc=/opt/rocm-5.4.3/bin/hipcc --with-hip-dir=/opt/rocm-5.4.3 COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" HIPOPTFLAGS="-g -O" --with-cuda=0 --with-hip=1 --with-precision=double --with-clanguage=c --download-kokkos --download-kokkos-kernels --download-hypre --download-magma --with-magma-fortran-bindings=0 --download-mfem --download-metis --with-strict-petscerrorcode PETSC_ARCH=arch-ci-linux-hip-double</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #1 MatMultAddKernel_SeqAIJHIPSPARSE() at /scratch/bsmith/petsc/src/mat/impls/aij/seq/seqhipsparse/aijhipsparse.hip.cpp:3131</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #2 MatMultAdd_SeqAIJHIPSPARSE() at /scratch/bsmith/petsc/src/mat/impls/aij/seq/seqhipsparse/aijhipsparse.hip.cpp:3004</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #3 MatMultAdd() at /scratch/bsmith/petsc/src/mat/interface/matrix.c:2770</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #4 MatInterpolateAdd() at /scratch/bsmith/petsc/src/mat/interface/matrix.c:8603</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #5 PCMGMCycle_Private() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:87</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #6 PCMGMCycle_Private() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:83</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #7 PCApply_MG_Internal() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:611</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #8 PCApply_MG() at /scratch/bsmith/petsc/src/ksp/pc/impls/mg/mg.c:633</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #9 PCApply() at /scratch/bsmith/petsc/src/ksp/pc/interface/precon.c:498</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #10 KSP_PCApply() at /scratch/bsmith/petsc/include/petsc/private/kspimpl.h:383</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #11 KSPSolve_Richardson() at /scratch/bsmith/petsc/src/ksp/ksp/impls/rich/rich.c:106</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #12 KSPSolve_Private() at /scratch/bsmith/petsc/src/ksp/ksp/interface/itfunc.c:906</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #13 KSPSolve() at /scratch/bsmith/petsc/src/ksp/ksp/interface/itfunc.c:1079</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #14 main() at ex34.c:52</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-stretch:normal;font-size:16px;line-height:normal;font-family:Menlo;font-kerning:auto;font-variant-alternates:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-feature-settings:normal"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: PETSc Option Table entries:</span></p><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">  Dave,</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">    Trying to debug the 7% now, but having trouble running, as you see above.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><br><blockquote type="cite"><div>On Jan 19, 2024, at 3:02 PM, Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Thank you Barry and Junchao for these explanations. I'll turn on -log_view_gpu_time.<div><br></div><div>Do either of you have any thoughts regarding why the percentage of flop's being reported on the GPU is not 100% for MGSmooth Level {0,1,2} for this solver configuration?</div><div><br></div><div>This number should have nothing to do with timings as it reports the ratio of operations performed on the GPU and CPU, presumably obtained from PetscLogFlops() and PetscLogGpuFlops().</div><div><br></div><div>Cheers,</div><div>Dave</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 19 Jan 2024 at 11:39, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Try to also add -log_view_gpu_time, <a href="https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/" target="_blank">https://petsc.org/release/manualpages/Profiling/PetscLogGpuTime/</a><div><br clear="all"><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 19, 2024 at 11:35 AM Dave May <<a href="mailto:dave.mayhem23@gmail.com" target="_blank">dave.mayhem23@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I am trying to understand the logging information associated with the %flops-performed-on-the-gpu reported by -log_view when running </div><div>  src/ksp/ksp/tutorials/ex34<br></div><div>with the following options</div><div>-da_grid_x 192<br>-da_grid_y 192<br>-da_grid_z 192<br>-dm_mat_type seqaijhipsparse<br>-dm_vec_type seqhip<br>-ksp_max_it 10<br>-ksp_monitor<br>-ksp_type richardson<br>-ksp_view<br>-log_view<br>-mg_coarse_ksp_max_it 2<br>-mg_coarse_ksp_type richardson<br>-mg_coarse_pc_type none<br>-mg_levels_ksp_type richardson<br>-mg_levels_pc_type none<br>-options_left<br>-pc_mg_levels 3<br>-pc_mg_log<br>-pc_type mg<br></div><div><br></div><div>This config is not intended to actually solve the problem, rather it is a stripped down set of options designed to understand what parts of the smoothers are being executed on the GPU.</div><div><br></div><div>With respect to the log file attached, my first set of questions related to the data reported under "Event Stage 2: MG Apply".</div><div><br></div><div>[1] Why is the log littered with nan's?</div><div>* I don't understand how and why "GPU Mflop/s" should be reported as nan when a value is given for "GPU %F" (see MatMult for example).</div><div><br></div><div>* For events executed on the GPU, I assume the column "Time (sec)" relates to "CPU execute time", this would explain why we see a nan in "Time (sec)" for MatMult.</div><div>If my assumption is correct, how should I interpret the column "Flop (Max)" which is showing 1.92e+09? </div><div>I would assume of "Time (sec)" relates to the CPU then "Flop (Max)" should also relate to CPU and GPU flops would be logged in "GPU Mflop/s"<br></div><div><br></div><div>[2] More curious is that within "Event Stage 2: MG Apply" KSPSolve, MGSmooth Level 0, MGSmooth Level 1, MGSmooth Level 2 all report "GPU %F" as 93. I believe this value should be 100 as the smoother (and coarse grid solver) are configured as richardson(2)+none and thus should run entirely on the GPU. </div><div>Furthermore, when one inspects all events listed under "Event Stage 2: MG Apply" those events which do flops correctly report "GPU %F" as 100. </div><div>And the events showing "GPU %F" = 0 such as <br></div><div>  MatHIPSPARSCopyTo, VecCopy, VecSet, PCApply, DCtxSync</div><div>don't do any flops (on the CPU or GPU) - which is also correct (although non GPU events should show nan??)<br></div><div><br></div><div>Hence I am wondering what is the explanation for the missing 7% from "GPU %F" for KSPSolve and MGSmooth {0,1,2}??</div><div><br></div><div>Does anyone understand this -log_view, or can explain to me how to interpret it?</div><div><br></div><div>It could simply be that:</div><div>a) something is messed up with -pc_mg_log</div><div>b) something is messed up with the PETSc build</div><div>c) I am putting too much faith in -log_view and should profile the code differently.</div><div><br></div><div>Either way I'd really like to understand what is going on.</div><div><br></div><div><br></div><div>Cheers,</div><div>Dave</div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>