<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><blockquote type="cite" class="">I suggested years ago that -log_view automatically print useful information about the GPU setup (when GPUs are used) but everyone seemed comfortable with the lack of information so no one improved it.</blockquote><div class=""><br class=""></div>FWIW, PetscDeviceView() does a bit of what you want (it just dumps all of cuda/hipDeviceProp)<div class=""><br class=""><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class=""></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 22, 2022, at 12:55, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><br class=""> I suggested years ago that -log_view automatically print useful information about the GPU setup (when GPUs are used) but everyone seemed comfortable with the lack of information so no one improved it. I think for a small number of GPUs -log_view should just print details and for a larger number print some statistics (how many physical ones etc). Currently, it does not even print how many are used. I think requiring another option to get this basic information is a mistake, we already print a ton of background with -log_view it is just sad no background on the GPU usage.<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">On Jan 22, 2022, at 1:06 PM, Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:<br class=""><br class="">Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> writes:<br class=""><br class=""><blockquote type="cite" class="">On Sat, Jan 22, 2022 at 12:29 PM Jed Brown <<a href="mailto:jed@jedbrown.org" class="">jed@jedbrown.org</a>> wrote:<br class=""><br class=""><blockquote type="cite" class="">Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> writes:<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">VecPointwiseMult 402 1.0 2.9605e-01 3.6 1.05e+08 1.0 0.0e+00<br class=""></blockquote></blockquote></blockquote>0.0e+00<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.0e+00 0 0 0 0 0 5 1 0 0 0 22515 70608 0 0.00e+00<br class=""></blockquote></blockquote>0<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.00e+00 100<br class=""><blockquote type="cite" class="">VecScatterBegin 400 1.0 1.6791e-01 6.0 0.00e+00 0.0 3.7e+05<br class=""></blockquote></blockquote></blockquote>1.6e+04<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.0e+00 0 0 62 54 0 2 0100100 0 0 0 0 0.00e+00<br class=""></blockquote></blockquote>0<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.00e+00 0<br class=""><blockquote type="cite" class="">VecScatterEnd 400 1.0 1.0057e+00 7.0 0.00e+00 0.0 0.0e+00<br class=""></blockquote></blockquote></blockquote>0.0e+00<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.0e+00 0 0 0 0 0 5 0 0 0 0 0 0 0 0.00e+00<br class=""></blockquote></blockquote>0<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.00e+00 0<br class=""><blockquote type="cite" class="">PCApply 402 1.0 2.9638e-01 3.6 1.05e+08 1.0 0.0e+00<br class=""></blockquote></blockquote></blockquote>0.0e+00<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.0e+00 0 0 0 0 0 5 1 0 0 0 22490 70608 0 0.00e+00<br class=""></blockquote></blockquote>0<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">0.00e+00 100<br class=""><br class="">Most of the MatMult time is attributed to VecScatterEnd here. Can you<br class="">share a run of the same total problem size on 8 ranks (one rank per<br class=""></blockquote></blockquote>GPU)?<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class=""><br class=""></blockquote>attached. I ran out of memory with the same size problem so this is the<br class="">262K / GPU version.<br class=""></blockquote><br class="">How was this launched? Is it possible all 8 ranks were using the same GPU?<br class="">(Perf is that bad.)<br class=""><br class=""></blockquote><br class="">srun -n8 -N1 *--ntasks-per-gpu=1* --gpu-bind=closest ../ex13<br class="">-dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 2,2,2<br class="">-dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1<br class="">-dm_refine 6 -dm_view -dm_mat_type aijkokkos -dm_vec_type kokkos -pc_type<br class="">jacobi -log_view -ksp_view -use_gpu_aware_mpi true<br class=""></blockquote><br class="">I'm still worried because the results are so unreasonable. We should add an option like -view_gpu_busid that prints this information per rank.<br class=""><br class=""><a href="https://code.ornl.gov/olcf/hello_jobstep/-/blob/master/hello_jobstep.cpp" class="">https://code.ornl.gov/olcf/hello_jobstep/-/blob/master/hello_jobstep.cpp</a><br class=""><br class="">A single-process/single-GPU comparison would also be a useful point of comparison.<br class=""></blockquote><br class=""></div></div></blockquote></div><br class=""></div></body></html>