<div dir="ltr"><div dir="ltr">On Wed, Aug 5, 2020 at 1:09 PM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk">g.gibb@epcc.ed.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">



<div style="overflow-wrap: break-word;">
Hi,
<div><br>
</div>
<div>I used nvidia-smi before, essentially a kind of ’top’ for nvidia-gpus.</div>
<div><br>
</div>
<div>The log output I get is:</div></div></blockquote><div><br></div><div>You can see that all flops are done on the GPU by looking at the last column:</div><div><br></div><div><font face="monospace">Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   - GpuToCpu - GPU<br>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size   Count   Size  %F<br>---------------------------------------------------------------------------------------------------------------------------------------------------------------<br><br>--- Event Stage 0: Main Stage<br><br>VecDot                 4 1.0 7.4222e-05 1.0 1.96e+02 1.0 0.0e+00 0.0e+00 0.0e+00  0  8  0  0  0   0  8  0  0  0     3       3      0 0.00e+00    0 0.00e+00 100<br>VecNorm                1 1.0 5.4168e-05 1.0 7.30e+01 1.0 0.0e+00 0.0e+00 0.0e+00  0  3  0  0  0   0  3  0  0  0     1       1      0 0.00e+00    0 0.00e+00 100<br></font></div><div> </div><div>  Thanks,</div><div><br></div><div>     Matt</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">************************************************************************************************************************</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">***             WIDEN YOUR WINDOW TO 120 CHARACTERS.  Use 'enscript -r -fCourier9' to print this document            ***</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">************************************************************************************************************************</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">---------------------------------------------- PETSc Performance Summary: ----------------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                       WARNING!!!                       #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   This code was compiled with a debugging option.      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   To get timing results run ./configure                #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   using --with-debugging=no, the performance will      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   be generally two or three times faster.              #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                       WARNING!!!                       #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   This code was compiled with GPU support but you used #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   an MPI that's not GPU-aware, such Petsc had to copy  #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   data from GPU to CPU for MPI communication. To get   #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   meaningfull timing results, please use a GPU-aware   #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   MPI instead.                                         #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">/lustre/home/z04/gpsgibb/TPLS/petsc/share/petsc/examples/src/vec/vec/tests/./ex28 on a  named r2i7n0 with 1 processor, by gpsgibb Wed Aug  5 18:05:59 2020</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using Petsc Release Version 3.13.3, Jul 01, 2020 </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                         Max       Max/Min     Avg       Total</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Time (sec):           1.566e-01     1.000   1.566e-01</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Objects:              4.400e+01     1.000   4.400e+01</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Flop:                 2.546e+03     1.000   2.546e+03  2.546e+03</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Flop/sec:             1.626e+04     1.000   1.626e+04  1.626e+04</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Memory:               1.438e+05     1.000   1.438e+05  1.438e+05</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">MPI Messages:         0.000e+00     0.000   0.000e+00  0.000e+00</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">MPI Message Lengths:  0.000e+00     0.000   0.000e+00  0.000e+00</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">MPI Reductions:       0.000e+00     0.000</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                            e.g., VecAXPY() for real vectors of length N --> 2N flop</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                            and VecAXPY() for complex vectors of length N --> 8N flop</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Summary of Stages:   ----- Time ------  ----- Flop ------  --- Messages ---  -- Message Lengths --  -- Reductions --</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                        Avg     %Total     Avg     %Total    Count   %Total     Avg         %Total    Count   %Total</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures"> 0:      Main Stage: 1.5657e-01 100.0%  2.5460e+03 100.0%  0.000e+00   0.0%  0.000e+00        0.0%  0.000e+00   0.0%</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Phase summary info:</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Count: number of times phase was executed</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Time and Flop: Max - maximum over all processors</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                  Ratio - ratio of maximum to minimum over all processors</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Mess: number of messages sent</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   AvgLen: average message length (bytes)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Reduct: number of global reductions</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Global: entire computation</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      %T - percent time in this phase         %F - percent flop in this phase</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      %M - percent messages in this phase     %L - percent message lengths in this phase</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      %R - percent reductions in this phase</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   CpuToGpu Count: total number of CPU to GPU copies per processor</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   GpuToCpu Count: total number of GPU to CPU copies per processor</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">   GPU %F: percent flops on GPU in this event</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                       WARNING!!!                       #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   This code was compiled with a debugging option.      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   To get timing results run ./configure                #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   using --with-debugging=no, the performance will      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   be generally two or three times faster.              #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Event                Count      Time (sec)     Flop                              --- Global ---  --- Stage ----  Total   GPU    - CpuToGpu -   - GpuToCpu - GPU</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">                   Max Ratio  Max     Ratio   Max  Ratio  Mess   AvgLen  Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s Mflop/s Count   Size   Count   Size  %F</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">--- Event Stage 0: Main Stage</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecDot                 4 1.0 7.4222e-05 1.0 1.96e+02 1.0 0.0e+00 0.0e+00 0.0e+00  0  8  0  0  0   0  8  0  0  0     3       3      0 0.00e+00    0 0.00e+00 100</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecNorm                1 1.0 5.4168e-05 1.0 7.30e+01 1.0 0.0e+00 0.0e+00 0.0e+00  0  3  0  0  0   0  3  0  0  0     1       1      0 0.00e+00    0 0.00e+00 100</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecSet                83 1.0 9.0480e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecAssemblyBegin       1 1.0 2.7206e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecAssemblyEnd         1 1.0 2.6403e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecSetRandom           1 1.0 1.5260e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecReduceArith        52 1.0 1.1307e-03 1.0 2.28e+03 1.0 0.0e+00 0.0e+00 0.0e+00  1 89  0  0  0   1 89  0  0  0     2       2      2 4.00e-04    0 0.00e+00 100</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecReduceComm          4 1.0 3.4969e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecReduceBegin         1 1.0 2.5639e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecReduceEnd           1 1.0 2.5495e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecCUDACopyTo          2 1.0 1.7550e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      2 4.00e-04    0 0.00e+00  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">VecCUDACopyFrom       42 1.0 3.7747e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0       0      0 0.00e+00   42 8.40e-03  0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Memory usage is given in bytes:</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Object Type          Creations   Destructions     Memory  Descendants' Mem.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Reports information only for process 0.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">--- Event Stage 0: Main Stage</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">              Vector    42             42        75264     0.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">         PetscRandom     1              1          646     0.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">              Viewer     1              0            0     0.</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">========================================================================================================================</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Average time to get PetscTime(): 3.67989e-08</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">#PETSc Option Table entries:</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-log_view</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-use_gpu_aware_mpi 0</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-vec_type cuda</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">#End of PETSc Option Table entries</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Compiled without FORTRAN kernels</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Compiled with full precision matrices (default)</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Configure options: CC=nvcc FC=mpif90 CXX=mpicxx --prefix=/lustre/home/z04/gpsgibb/TPLS/petsc --with-cudac=nvcc --with-cuda=1 --with-mpi-dir= --with-batch</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Libraries compiled on 2020-07-31 14:46:25 on r2i7n0 </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Machine characteristics: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using PETSc directory: /lustre/home/z04/gpsgibb/TPLS/petsc</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using PETSc arch: </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using C compiler: nvcc  -g  -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using Fortran compiler: mpif90  -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g     -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using include paths: -I/lustre/home/z04/gpsgibb/TPLS/petsc/include -I/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/include -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using C linker: nvcc</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using Fortran linker: mpif90</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">Using libraries: -L/lustre/home/z04/gpsgibb/TPLS/petsc/lib -L/lustre/home/z04/gpsgibb/TPLS/petsc/lib -lpetsc -L/lustre/sw/intel/compilers_and_libraries_2019.0.117/linux/mkl -L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/lib64
 -L/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/lib -L/opt/hpe/hpc/mpt/mpt-2.22/lib -L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/20.5/math_libs/10.2/lib64 -L/lustre/sw/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -L/lustre/sw/gcc/6.3.0/lib64 -L/lustre/sw/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64
 -L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/bin -L/lustre/sw/gcc/6.3.0/lib -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lX11 -lcufft -lcublas -lcudart -lcusparse -lcusolver -lcuda -lmpi++ -lmpi -lstdc++ -ldl -lpthread -lmpi -lgfortran -lm
 -lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">-----------------------------------------</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                       WARNING!!!                       #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   This code was compiled with GPU support but you used #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   an MPI that's not GPU-aware, such Petsc had to copy  #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   data from GPU to CPU for MPI communication. To get   #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   meaningfull timing results, please use a GPU-aware   #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   MPI instead.                                         #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                       WARNING!!!                       #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   This code was compiled with a debugging option.      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   To get timing results run ./configure                #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   using --with-debugging=no, the performance will      #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #   be generally two or three times faster.              #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      #                                                        #</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">      ##########################################################</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"></span><br>
</div>
<div>
<div style="overflow-wrap: break-word;">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On 5 Aug 2020, at 17:58, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr"></div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 5, 2020 at 12:47 PM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" target="_blank">g.gibb@epcc.ed.ac.uk</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">Hi Matt,
<div><br>
</div>
<div>It runs, however it doesn’t produce any output, and I have no way of checking to see if it actually ran on the GPU. It was run with:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) How did you check last time?</div>
<div><br>
</div>
<div>2) You can check using -log_view</div>
<div><br>
</div>
<div>  Thanks,</div>
<div><br>
</div>
<div>     Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
<div>Cheers,</div>
<div><br>
</div>
<div>Gordon</div>
<div><br>
<div>
<div style="overflow-wrap: break-word;">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On 5 Aug 2020, at 17:10, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr">On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" target="_blank">g.gibb@epcc.ed.ac.uk</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">Hi,
<div><br>
</div>
<div>I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" target="_blank">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable
 against this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="font-family:"Hack Nerd Font"">-use_gpu_aware_mpi 0</span></div>
<div><br>
</div>
<div>When running this, in the .petscrc I put</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">-dm_vec_type cuda</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">-dm_mat_type aijcusparse</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs).
 I have also ensured I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div><br>
</div>
<div>When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">Wed Aug  5 13:10:23 2020       </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|===============================+======================+======================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|   0  Tesla V100-SXM2...  On   | 00000000:1A:00.0 Off |                  Off |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| N/A   43C    P0    64W / 300W |    490MiB / 16160MiB |      0%      Default |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-------------------------------+----------------------+----------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures">                                                                               </span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| Processes:                                                       GPU Memory |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|  GPU       PID   Type   Process name                             Usage      |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|=============================================================================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|    0     33712      C   .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x   479MiB |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
 I then tried to run the example with the <span>-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least a warning if invalid arguments were passed to it.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
 regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">What am I doing wrong?</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Lets step back to a simpler thing so we can make sure your configuration is correct. Can you run the 2_cuda test from</div>
<div>src/vec/vec/tests/ex28.c ? Does it execute on your GPU?</div>
<div><br>
</div>
<div>  Thanks,</div>
<div><br>
</div>
<div>    Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
<div>
<div><span style="font-variant-ligatures:no-common-ligatures">Thanks in advance,</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">Gordon</span></div>
<div>
<div style="overflow-wrap: break-word;">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<br>
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
--<span> </span><br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
--<span> </span><br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>