<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi,
<div class=""><br class="">
</div>
<div class="">I used nvidia-smi before, essentially a kind of ’top’ for nvidia-gpus.</div>
<div class=""><br class="">
</div>
<div class="">The log output I get is:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">************************************************************************************************************************</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">************************************************************************************************************************</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">---------------------------------------------- PETSc Performance Summary: ----------------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # WARNING!!! #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # This code was compiled with a debugging option. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # To get timing results run ./configure #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # using --with-debugging=no, the performance will #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # be generally two or three times faster. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # WARNING!!! #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # This code was compiled with GPU support but you used #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # an MPI that's not GPU-aware, such Petsc had to copy #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # data from GPU to CPU for MPI communication. To get #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # meaningfull timing results, please use a GPU-aware #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # MPI instead. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">/lustre/home/z04/gpsgibb/TPLS/petsc/share/petsc/examples/src/vec/vec/tests/./ex28 on a named r2i7n0 with 1 processor, by gpsgibb Wed Aug 5 18:05:59 2020</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using Petsc Release Version 3.13.3, Jul 01, 2020 </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Max Max/Min Avg Total</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Time (sec): 1.566e-01 1.000 1.566e-01</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Objects: 4.400e+01 1.000 4.400e+01</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Flop: 2.546e+03 1.000 2.546e+03 2.546e+03</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Flop/sec: 1.626e+04 1.000 1.626e+04 1.626e+04</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Memory: 1.438e+05 1.000 1.438e+05 1.438e+05</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">MPI Messages: 0.000e+00 0.000 0.000e+00 0.000e+00</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">MPI Message Lengths: 0.000e+00 0.000 0.000e+00 0.000e+00</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">MPI Reductions: 0.000e+00 0.000</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> e.g., VecAXPY() for real vectors of length N --> 2N flop</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> and VecAXPY() for complex vectors of length N --> 8N flop</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Avg %Total Avg %Total Count %Total Avg %Total Count %Total</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> 0: Main Stage: 1.5657e-01 100.0% 2.5460e+03 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">See the 'Profiling' chapter of the users' manual for details on interpreting output.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Phase summary info:</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Count: number of times phase was executed</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Time and Flop: Max - maximum over all processors</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Ratio - ratio of maximum to minimum over all processors</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Mess: number of messages sent</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> AvgLen: average message length (bytes)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Reduct: number of global reductions</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Global: entire computation</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> %T - percent time in this phase %F - percent flop in this phase</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> %M - percent messages in this phase %L - percent message lengths in this phase</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> %R - percent reductions in this phase</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> GPU Mflop/s: 10e-6 * (sum of flop on GPU over all processors)/(max GPU time over all processors)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> CpuToGpu Count: total number of CPU to GPU copies per processor</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> CpuToGpu Size (Mbytes): 10e-6 * (total size of CPU to GPU copies per processor)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> GpuToCpu Count: total number of GPU to CPU copies per processor</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> GpuToCpu Size (Mbytes): 10e-6 * (total size of GPU to CPU copies per processor)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> GPU %F: percent flops on GPU in this event</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # WARNING!!! #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # This code was compiled with a debugging option. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # To get timing results run ./configure #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # using --with-debugging=no, the performance will #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # be generally two or three times faster. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Event Count Time (sec) Flop --- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">--- Event Stage 0: Main Stage</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecDot 4 1.0 7.4222e-05 1.0 1.96e+02 1.0 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 8 0 0 0 3 3 0 0.00e+00 0 0.00e+00 100</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecNorm 1 1.0 5.4168e-05 1.0 7.30e+01 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 1 1 0 0.00e+00 0 0.00e+00 100</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecSet 83 1.0 9.0480e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecAssemblyBegin 1 1.0 2.7206e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecAssemblyEnd 1 1.0 2.6403e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecSetRandom 1 1.0 1.5260e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecReduceArith 52 1.0 1.1307e-03 1.0 2.28e+03 1.0 0.0e+00 0.0e+00 0.0e+00 1 89 0 0 0 1 89 0 0 0 2 2 2 4.00e-04 0 0.00e+00 100</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecReduceComm 4 1.0 3.4969e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecReduceBegin 1 1.0 2.5639e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecReduceEnd 1 1.0 2.5495e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecCUDACopyTo 2 1.0 1.7550e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 2 4.00e-04 0 0.00e+00 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">VecCUDACopyFrom 42 1.0 3.7747e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 42 8.40e-03 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">---------------------------------------------------------------------------------------------------------------------------------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Memory usage is given in bytes:</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Object Type Creations Destructions Memory Descendants' Mem.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Reports information only for process 0.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">--- Event Stage 0: Main Stage</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Vector 42 42 75264 0.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> PetscRandom 1 1 646 0.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> Viewer 1 0 0 0.</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">========================================================================================================================</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Average time to get PetscTime(): 3.67989e-08</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">#PETSc Option Table entries:</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-log_view</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-use_gpu_aware_mpi 0</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-vec_type cuda</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">#End of PETSc Option Table entries</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Compiled without FORTRAN kernels</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Compiled with full precision matrices (default)</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Configure options: CC=nvcc FC=mpif90 CXX=mpicxx --prefix=/lustre/home/z04/gpsgibb/TPLS/petsc --with-cudac=nvcc --with-cuda=1 --with-mpi-dir= --with-batch</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-----------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Libraries compiled on 2020-07-31 14:46:25 on r2i7n0 </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Machine characteristics: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using PETSc directory: /lustre/home/z04/gpsgibb/TPLS/petsc</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using PETSc arch: </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-----------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using C compiler: nvcc -g -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using Fortran compiler: mpif90 -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-----------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using include paths: -I/lustre/home/z04/gpsgibb/TPLS/petsc/include -I/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/include -I/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/include</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-----------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using C linker: nvcc</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using Fortran linker: mpif90</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Using libraries: -L/lustre/home/z04/gpsgibb/TPLS/petsc/lib -L/lustre/home/z04/gpsgibb/TPLS/petsc/lib -lpetsc -L/lustre/sw/intel/compilers_and_libraries_2019.0.117/linux/mkl -L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/lib64
-L/lustre/home/z04/gpsgibb/TPLS/petsc-3.13.3/lib -L/opt/hpe/hpc/mpt/mpt-2.22/lib -L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/20.5/math_libs/10.2/lib64 -L/lustre/sw/gcc/6.3.0/lib/gcc/x86_64-pc-linux-gnu/6.3.0 -L/lustre/sw/gcc/6.3.0/lib64 -L/lustre/sw/intel/compilers_and_libraries_2019.0.117/linux/mkl/lib/intel64
-L/lustre/sw/nvidia/hpcsdk/Linux_x86_64/cuda/10.2/bin -L/lustre/sw/gcc/6.3.0/lib -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread -lX11 -lcufft -lcublas -lcudart -lcusparse -lcusolver -lcuda -lmpi++ -lmpi -lstdc++ -ldl -lpthread -lmpi -lgfortran -lm
-lgfortran -lm -lgcc_s -lquadmath -lstdc++ -ldl</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-----------------------------------------</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # WARNING!!! #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # This code was compiled with GPU support but you used #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # an MPI that's not GPU-aware, such Petsc had to copy #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # data from GPU to CPU for MPI communication. To get #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # meaningfull timing results, please use a GPU-aware #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # MPI instead. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # WARNING!!! #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # This code was compiled with a debugging option. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # To get timing results run ./configure #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # using --with-debugging=no, the performance will #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # be generally two or three times faster. #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> # #</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> ##########################################################</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8); min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class="">
</div>
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2020, at 17:58, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div dir="ltr" class=""></div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 5, 2020 at 12:47 PM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" class="">g.gibb@epcc.ed.ac.uk</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">Hi Matt,
<div class=""><br class="">
</div>
<div class="">It runs, however it doesn’t produce any output, and I have no way of checking to see if it actually ran on the GPU. It was run with:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0</span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">1) How did you check last time?</div>
<div class=""><br class="">
</div>
<div class="">2) You can check using -log_view</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">Gordon</div>
<div class=""><br class="">
<div class="">
<div style="word-wrap: break-word;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
Tel: +44 131 651 3459</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2020, at 17:10, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:</div>
<br class="">
<div class="">
<div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="ltr" class="">On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" target="_blank" class="">g.gibb@epcc.ed.ac.uk</a>> wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">Hi,
<div class=""><br class="">
</div>
<div class="">I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" target="_blank" class="">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable
against this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="font-family: "Hack Nerd Font";" class="">-use_gpu_aware_mpi 0</span></div>
<div class=""><br class="">
</div>
<div class="">When running this, in the .petscrc I put</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">-dm_vec_type cuda</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">-dm_mat_type aijcusparse</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class="">as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs).
I have also ensured I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div class=""><br class="">
</div>
<div class="">When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Wed Aug 5 13:10:23 2020 </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|===============================+======================+======================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| 0 Tesla V100-SXM2... On | 00000000:1A:00.0 Off | Off |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| N/A 43C P0 64W / 300W | 490MiB / 16160MiB | 0% Default |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| Processes: GPU Memory |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| GPU PID Type Process name Usage |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|=============================================================================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| 0 33712 C .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x 479MiB |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
I then tried to run the example with the <span class="">-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least a warning if invalid arguments were passed to it.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">What am I doing wrong?</span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Lets step back to a simpler thing so we can make sure your configuration is correct. Can you run the 2_cuda test from</div>
<div class="">src/vec/vec/tests/ex28.c ? Does it execute on your GPU?</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">
<div class="">
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Thanks in advance,</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Gordon</span></div>
<div class="">
<div style="word-wrap: break-word;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
Tel: +44 131 651 3459</div>
</div>
</div>
<br class="">
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class=""> </span><br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div dir="ltr" class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>