[petsc-users] Code (possibly) not running on GPU with CUDA

[Matthew Knepley]

On Wed, Aug 5, 2020 at 11:24 AM GIBB Gordon <g.gibb at epcc.ed.ac.uk> wrote:

> Hi,
>
> I’ve built PETSc with NVIDIA support for our GPU machine (
> https://cirrus.readthedocs.io/en/master/user-guide/gpu.html), and then
> compiled our executable against this PETSc (using version 3.13.3). I should
> add that the MPI on our system is not GPU-aware so I have to use -use_gpu_aware_mpi
> 0
>
> When running this, in the .petscrc I put
>
> -dm_vec_type cuda
> -dm_mat_type aijcusparse
>
> as is suggested on the PETSc GPU page (
> https://www.mcs.anl.gov/petsc/features/gpus.html) to enable CUDA for DMs
> (all our PETSc data structures are with DMs). I have also ensured I'm using
> the jacobi preconditioner so that it definitely runs on the GPU (again,
> according to the PETSc GPU page).
>
> When I run this, I note that the GPU seems to have memory allocated on it
> from my executable, however seems to be doing no computation:
>
> Wed Aug  5 13:10:23 2020
>
> +-----------------------------------------------------------------------------+
> | NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2
>     |
>
> |-------------------------------+----------------------+----------------------+
> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.
> ECC |
> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute
> M. |
>
> |===============================+======================+======================|
> |   0  Tesla V100-SXM2...  On   | 00000000:1A:00.0 Off |
> Off |
> | N/A   43C    P0    64W / 300W |    490MiB / 16160MiB |      0%
> Default |
>
> +-------------------------------+----------------------+----------------------+
>
>
>
>
> +-----------------------------------------------------------------------------+
> | Processes:                                                       GPU
> Memory |
> |  GPU       PID   Type   Process name                             Usage
>     |
>
> |=============================================================================|
> |    0     33712      C   .../z04/gpsgibb/TPLS/TPLS-GPU/./twophase.x
> 479MiB |
>
> +-----------------------------------------------------------------------------+
>
> I then ran the same example but without the -dm_vec_type cuda,
> -dm_mat_type aijcusparse arguments, and I found the same behaviour (479MB
> allocated on the GPU, 0% GPU utilisation).
>
> In both cases the runtime of the example are near identical, suggesting
> that both are essentially the same run.
>
> As a further test I compiled PETSc without CUDA support and ran the same
> example again, and found the same runtime as with the GPUs, and (as
> expected) no GPU memory allocated. I then tried to run the example with the -dm_vec_type
> cuda, -dm_mat_type aijcusparse arguments and it ran without complaint. I
> would have expected it to throw an error or at least a warning if invalid
> arguments were passed to it.
>
> All this suggests to me that PETSc is ignoring my requests to use the
> GPUs. For the GPU-aware PETSc it seems to allocate memory on the GPUs but
> perform no calculations on them, regardless of whether I requested it to
> use the GPUs or not. On non-GPU-aware PETSc it accepts my requests to use
> the GPUs, but does not throw an error.
>
> What am I doing wrong?
>

Lets step back to a simpler thing so we can make sure your configuration is
correct. Can you run the 2_cuda test from
src/vec/vec/tests/ex28.c ? Does it execute on your GPU?

  Thanks,

    Matt


> Thanks in advance,
>
> Gordon
> -----------------------------------------------
> Dr Gordon P S Gibb
> EPCC, The University of Edinburgh
> Tel: +44 131 651 3459
>
> The University of Edinburgh is a charitable body, registered in Scotland,
> with registration number SC005336.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200805/ca243071/attachment.html>