<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Matt,
<div class=""><br class="">
</div>
<div class="">It runs, however it doesn’t produce any output, and I have no way of checking to see if it actually ran on the GPU. It was run with:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; color: rgb(36, 45, 62); background-color: rgba(255, 255, 255, 0.8);" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">srun -n 1 ./ex28 -vec_type cuda -use_gpu_aware_mpi 0</span></div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">Gordon</div>
<div class=""><br class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2020, at 17:10, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<div dir="ltr" class="">On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" class="">g.gibb@epcc.ed.ac.uk</a>> wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">Hi,
<div class=""><br class="">
</div>
<div class="">I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" target="_blank" class="">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable
against this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="font-family: "Hack Nerd Font";" class="">-use_gpu_aware_mpi 0</span></div>
<div class=""><br class="">
</div>
<div class="">When running this, in the .petscrc I put</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">-dm_vec_type cuda</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">-dm_mat_type aijcusparse</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class="">as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs).
I have also ensured I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div class=""><br class="">
</div>
<div class="">When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">Wed Aug 5 13:10:23 2020 </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|===============================+======================+======================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| 0 Tesla V100-SXM2... On | 00000000:1A:00.0 Off | Off |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| N/A 43C P0 64W / 300W | 490MiB / 16160MiB | 0% Default |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class=""> </span><br class="">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| Processes: GPU Memory |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| GPU PID Type Process name Usage |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">|=============================================================================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">| 0 33712 C .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x 479MiB |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures;" class="">+-----------------------------------------------------------------------------+</span></div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
I then tried to run the example with the <span style="" class="">-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least a warning if invalid arguments were passed to
it.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">What am I doing wrong?</span></div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Lets step back to a simpler thing so we can make sure your configuration is correct. Can you run the 2_cuda test from</div>
<div class="">src/vec/vec/tests/ex28.c ? Does it execute on your GPU?</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">
<div style="word-wrap: break-word;" class="">
<div class="">
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Thanks in advance,</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">Gordon</span></div>
<div class="">
<div style="word-wrap: break-word;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
Tel: +44 131 651 3459</div>
</div>
</div>
<br class="">
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
--<span class="Apple-converted-space"> </span><br class="">
<div dir="ltr" class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>