<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I do not have DMSetFromOptions in the code, so this may well be the issue<br class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
Tel: +44 131 651 3459</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2020, at 18:48, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
Gordon,
<div class=""><br class="">
</div>
<div class=""> Do you have a call to DMSetFromOptions()? </div>
<div class=""><br class="">
</div>
<div class=""> Barry</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 5, 2020, at 10:24 AM, GIBB Gordon <<a href="mailto:g.gibb@epcc.ed.ac.uk" class="">g.gibb@epcc.ed.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi,
<div class=""><br class="">
</div>
<div class="">I’ve built PETSc with NVIDIA support for our GPU machine (<a href="https://cirrus.readthedocs.io/en/master/user-guide/gpu.html" class="">https://cirrus.readthedocs.io/en/master/user-guide/gpu.html</a>), and then compiled our executable against
this PETSc (using version 3.13.3). I should add that the MPI on our system is not GPU-aware so I have to use <span style="font-family: "Hack Nerd Font";" class="">-use_gpu_aware_mpi 0</span></div>
<div class=""><br class="">
</div>
<div class="">When running this, in the .petscrc I put</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-dm_vec_type cuda</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">-dm_mat_type aijcusparse</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class="">as is suggested on the PETSc GPU page (<a href="https://www.mcs.anl.gov/petsc/features/gpus.html" class="">https://www.mcs.anl.gov/petsc/features/gpus.html</a>) to enable CUDA for DMs (all our PETSc data structures are with DMs). I have also ensured
I'm using the jacobi preconditioner so that it definitely runs on the GPU (again, according to the PETSc GPU page).</div>
<div class=""><br class="">
</div>
<div class="">When I run this, I note that the GPU seems to have memory allocated on it from my executable, however seems to be doing no computation:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">Wed Aug 5 13:10:23 2020 </span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2 |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">|-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">|===============================+======================+======================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| 0 Tesla V100-SXM2... On | 00000000:1A:00.0 Off | Off |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| N/A 43C P0 64W / 300W | 490MiB / 16160MiB | 0% Default |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">+-------------------------------+----------------------+----------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font"; min-height: 14px;" class="">
<span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder">
</div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">+-----------------------------------------------------------------------------+</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| Processes: GPU Memory |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| GPU PID Type Process name Usage |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">|=============================================================================|</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">| 0 33712 C .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x 479MiB |</span></div>
<div style="margin: 0px; font-stretch: normal; line-height: normal; font-family: "Hack Nerd Font";" class="">
<span style="font-variant-ligatures: no-common-ligatures" class="">+-----------------------------------------------------------------------------+</span></div>
</div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">I then ran the same example but without the -dm_vec_type cuda, -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB allocated on the GPU, 0% GPU utilisation).</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">In both cases the runtime of the example are near identical, suggesting that both are essentially the same run.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">As a further test I compiled PETSc without CUDA support and ran the same example again, and found the same runtime as with the GPUs, and (as expected) no GPU memory allocated.
I then tried to run the example with the <span style="caret-color: rgb(0, 0, 0);" class="">-dm_vec_type cuda, -dm_mat_type aijcusparse arguments</span> and it ran without complaint. I would have expected it to throw an error or at least a warning if invalid
arguments were passed to it.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">All this suggests to me that PETSc is ignoring my requests to use the GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but perform no calculations on them,
regardless of whether I requested it to use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use the GPUs, but does not throw an error.</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">What am I doing wrong?</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Thanks in advance,</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Gordon</span></div>
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
-----------------------------------------------<br class="">
Dr Gordon P S Gibb<br class="">
EPCC, The University of Edinburgh</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Tel: +44 131 651 3459</div>
</div>
</div>
<br class="">
</div>
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>