<div dir="ltr"><div dir="ltr">On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk">g.gibb@epcc.ed.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
Hi,
<div><br>
</div>
<div>I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" target="_blank">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable against
this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="color:rgb(0,0,0);font-family:"Hack Nerd Font"">-use_gpu_aware_mpi 0</span></div>
<div><br>
</div>
<div>When running this, in the .petscrc I put</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">-dm_vec_type cuda</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">-dm_mat_type aijcusparse</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs). I have also ensured
I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div><br>
</div>
<div>When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">Wed Aug 5 13:10:23 2020 </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">|===============================+======================+======================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| 0 Tesla V100-SXM2... On | 00000000:1A:00.0 Off | Off |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| N/A 43C P0 64W / 300W | 490MiB / 16160MiB | 0% Default |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">+-------------------------------+----------------------+----------------------+</span></div>
<p style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0);min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"> </span><br>
</p>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| Processes: GPU Memory |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| GPU PID Type Process name Usage |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">|=============================================================================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">| 0 33712 C .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x 479MiB |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(0,0,0)">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
I then tried to run the example with the <span style="color:rgb(0,0,0)">-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least
a warning if invalid arguments were passed to it.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">What am I doing wrong?</span></div></div></div></blockquote><div><br></div><div>Lets step back to a simpler thing so we can make sure your configuration is correct. Can you run the 2_cuda test from</div><div>src/vec/vec/tests/ex28.c ? Does it execute on your GPU?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div>
<div><span style="font-variant-ligatures:no-common-ligatures">Thanks in advance,</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">Gordon</span></div>
<div>
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<br>
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>