<div dir="ltr"><div dir="ltr"></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 5, 2020 at 12:47 PM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk">g.gibb@epcc.ed.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
Hi Matt,
<div><br>
</div>
<div>It runs, however it doesn’t produce any output, and I have no way of checking to see if it actually ran on the GPU. It was run with:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";color:rgb(36,45,62);background-color:rgba(255,255,255,0.8)">
<span style="font-variant-ligatures:no-common-ligatures">srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0</span></div></div></div></blockquote><div><br></div><div>1) How did you check last time?</div><div><br></div><div>2) You can check using -log_view</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">
<div>Cheers,</div>
<div><br>
</div>
<div>Gordon</div>
<div><br>
<div>
<div style="word-wrap:break-word">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On 5 Aug 2020, at 17:10, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="ltr">On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" target="_blank">g.gibb@epcc.ed.ac.uk</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">Hi,
<div><br>
</div>
<div>I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" target="_blank">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable
against this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="font-family:"Hack Nerd Font"">-use_gpu_aware_mpi 0</span></div>
<div><br>
</div>
<div>When running this, in the .petscrc I put</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">-dm_vec_type cuda</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">-dm_mat_type aijcusparse</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs).
I have also ensured I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div><br>
</div>
<div>When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div><br>
</div>
<div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">Wed Aug 5 13:10:23 2020 </span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|===============================+======================+======================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| 0 Tesla V100-SXM2... On | 00000000:1A:00.0 Off | Off |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| N/A 43C P0 64W / 300W | 490MiB / 16160MiB | 0% Default |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-------------------------------+----------------------+----------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font";min-height:14px">
<span style="font-variant-ligatures:no-common-ligatures"> </span><br>
</div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| Processes: GPU Memory |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| GPU PID Type Process name Usage |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">|=============================================================================|</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">| 0 33712 C .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x 479MiB |</span></div>
<div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Hack Nerd Font"">
<span style="font-variant-ligatures:no-common-ligatures">+-----------------------------------------------------------------------------+</span></div>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
I then tried to run the example with the <span>-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least a warning if invalid arguments were passed to
it.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">What am I doing wrong?</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Lets step back to a simpler thing so we can make sure your configuration is correct. Can you run the 2_cuda test from</div>
<div>src/vec/vec/tests/ex28.c ? Does it execute on your GPU?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word">
<div>
<div><span style="font-variant-ligatures:no-common-ligatures">Thanks in advance,</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures">Gordon</span></div>
<div>
<div style="word-wrap:break-word">
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
-----------------------------------------------<br>
Dr Gordon P S Gibb<br>
EPCC, The University of Edinburgh</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Tel: +44 131 651 3459</div>
</div>
</div>
<br>
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
--<span> </span><br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>