<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 22, 2022 at 12:29 PM Jed Brown <<a href="mailto:jed@jedbrown.org">jed@jedbrown.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> writes:<br>
<br>
>><br>
>><br>
>><br>
>> > VecPointwiseMult 402 1.0 2.9605e-01 3.6 1.05e+08 1.0 0.0e+00 0.0e+00<br>
>> 0.0e+00 0 0 0 0 0 5 1 0 0 0 22515 70608 0 0.00e+00 0<br>
>> 0.00e+00 100<br>
>> > VecScatterBegin 400 1.0 1.6791e-01 6.0 0.00e+00 0.0 3.7e+05 1.6e+04<br>
>> 0.0e+00 0 0 62 54 0 2 0100100 0 0 0 0 0.00e+00 0<br>
>> 0.00e+00 0<br>
>> > VecScatterEnd 400 1.0 1.0057e+00 7.0 0.00e+00 0.0 0.0e+00 0.0e+00<br>
>> 0.0e+00 0 0 0 0 0 5 0 0 0 0 0 0 0 0.00e+00 0<br>
>> 0.00e+00 0<br>
>> > PCApply 402 1.0 2.9638e-01 3.6 1.05e+08 1.0 0.0e+00 0.0e+00<br>
>> 0.0e+00 0 0 0 0 0 5 1 0 0 0 22490 70608 0 0.00e+00 0<br>
>> 0.00e+00 100<br>
>><br>
>> Most of the MatMult time is attributed to VecScatterEnd here. Can you<br>
>> share a run of the same total problem size on 8 ranks (one rank per GPU)?<br>
>><br>
>><br>
> attached. I ran out of memory with the same size problem so this is the<br>
> 262K / GPU version.<br>
<br>
How was this launched? Is it possible all 8 ranks were using the same GPU? (Perf is that bad.)<br></blockquote><div><br></div><div>srun -n8 -N1 <b>--ntasks-per-gpu=1</b> --gpu-bind=closest ../ex13 -dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 2,2,2 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_refine 6 -dm_view -dm_mat_type aijkokkos -dm_vec_type kokkos -pc_type jacobi -log_view -ksp_view -use_gpu_aware_mpi true</div><div><br></div><div>+ a large .petscrc file </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
>> From the other log file (10x bigger problem)<br>
>><br>
>><br>
> ????<br>
<br>
You had attached two files and the difference seemed to be that the second was 10x more dofs/rank.<br></blockquote><div><br></div><div>I am refining a cube so it goes by 8x. </div><div> </div><div>jac_out_<font color="#ff0000">001</font>_kokkos_Crusher_<font color="#ff9900">6</font>_<font color="#00ff00">1</font>_notpl.txt<br></div><div><br></div><div><font color="#ff0000">number of nodes</font></div><div><font color="#ff9900">number of refinements </font></div><div><font color="#00ff00">number of process per GPU</font></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> --- Event Stage 2: KSP Solve only<br>
><br>
> MatMult 400 1.0 8.8003e+00 1.1 1.06e+11 1.0 2.2e+04 8.5e+04 0.0e+00 2 55 61 54 0 70 91100100 0 95058 132242 0 0.00e+00 0 0.00e+00 100<br>
> MatView 2 1.0 1.1643e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> KSPSolve 2 1.0 1.2540e+01 1.0 1.17e+11 1.0 2.2e+04 8.5e+04 1.2e+03 3 60 61 54 60 100100100100100 73592 116796 0 0.00e+00 0 0.00e+00 100<br>
> SFPack 400 1.0 1.8276e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> SFUnpack 400 1.0 6.2653e-05 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> VecTDot 802 1.0 1.3551e+00 1.2 3.36e+09 1.0 0.0e+00 0.0e+00 8.0e+02 0 2 0 0 40 10 3 0 0 67 19627 52599 0 0.00e+00 0 0.00e+00 100<br>
> VecNorm 402 1.0 9.0151e-01 2.2 1.69e+09 1.0 0.0e+00 0.0e+00 4.0e+02 0 1 0 0 20 5 1 0 0 33 14788 125477 0 0.00e+00 0 0.00e+00 100<br>
> VecCopy 4 1.0 7.3905e-03 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> VecSet 4 1.0 3.1814e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> VecAXPY 800 1.0 8.2617e-01 1.0 3.36e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 7 3 0 0 0 32112 61644 0 0.00e+00 0 0.00e+00 100<br>
> VecAYPX 398 1.0 8.1525e-01 1.6 1.67e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 5 1 0 0 0 16190 20689 0 0.00e+00 0 0.00e+00 100<br>
> VecPointwiseMult 402 1.0 3.5694e-01 1.0 8.43e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 1 0 0 0 18675 38633 0 0.00e+00 0 0.00e+00 100<br>
> VecScatterBegin 400 1.0 1.3391e+00 2.6 0.00e+00 0.0 2.2e+04 8.5e+04 0.0e+00 0 0 61 54 0 7 0100100 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> VecScatterEnd 400 1.0 1.3240e+00 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 9 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<br>
> PCApply 402 1.0 3.5712e-01 1.0 8.43e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 3 1 0 0 0 18665 38633 0 0.00e+00 0 0.00e+00 100<br>
</blockquote></div></div>