[petsc-users] A series of GPU questions

[Barry Smith]

   You might take a look at https://publications.anl.gov/anlpubs/2020/04/159190.pdf <https://publications.anl.gov/anlpubs/2020/04/159190.pdf>  in the introduction there is a short discussion about some of the 
"gotchas" when using multi-core CPUs connected to multiple GPUs; it is focused on an IBM Power/Nvidia GPU system but the same abstract issues
will arise on any similar system.

1) How many cores should share a GPU?   Generally 1 but there may be exceptions

2) Special memory utilization that can be copied to/from GPUs faster, it should be turned on?

3) Is it worth doing anything with the "extra" cores that are not accessing a GPU?   Probably not but there may be exceptions.

4) How to communicate between nodes with MPI, can one go directly from GPU to GPU and skip the CPU memory?

  Barry




> On Jun 9, 2020, at 7:51 PM, Junchao Zhang <junchao.zhang at gmail.com> wrote:
> 
> 
> 
> On Tue, Jun 9, 2020 at 7:11 PM GIBB Gordon <g.gibb at epcc.ed.ac.uk <mailto:g.gibb at epcc.ed.ac.uk>> wrote:
> Hi,
> 
> First of all, my apologies if this is not the appropriate list to send these questions to.
> 
> I’m one of the developers of TPLS (https://sourceforge.net/projects/tpls/ <https://sourceforge.net/projects/tpls/>), a Fortran code that uses PETSc, parallelised using DM vectors. It uses a mix of our own solvers, and PETSc’s Krylov solvers. At present it has been run on up to 25,000 MPI processes, although larger problem sizes should be able to scale beyond that.
> 
> With the awareness that more and more HPC machines now have one or more GPUs per node, and that upcoming machines that approach/achieve Exascale will be heterogeneous in nature, we are investigating whether it is worth using GPUs with TPLS, and if so, how to best do this.
> 
> I see that in principle all we’d need to do to is set some flags as described at https://www.mcs.anl.gov/petsc/features/gpus.html <https://www.mcs.anl.gov/petsc/features/gpus.html> to offload work onto the GPU, however I have some questions about doing this in practice:
> 
> The GPU machine I have access to has nodes with two 20 core CPUs and 4 NVIDIA GPUs (so 10 cores per GPU). We could use CUDA or OpenCL, and may well explore both of them. With TPLS being an MPI application, we would wish to use many processes (and nodes), not just a single process. How would we best split this problem up? 
> 
> Would we have 1 MPI process per GPU (so 4 per node), and then implement our own solvers either to also work on the GPU, or use OpenMP to make use of the 10 cores per GPU? If so, how would we specify to PETSc which GPU each process would use? 
> 
> Would we instead just have 40 (or perhaps slightly fewer) MPI processes all sharing the GPUs? Surely this would be inefficient, and would PETSc distribute the work across all 4 GPUs, or would every process end out using a single GPU?
> See https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service <https://docs.olcf.ornl.gov/systems/summit_user_guide.html#volta-multi-process-service>.  In some cases, we did see better performance with multiple mpi ranks/GPU than 1 rank/GPU. The optimal configuration depends on the code. Think two extremes:  One code with work done all on GPU and the other all on CPU. Probably you only need 1 mpi rank/node for the former, but full ranks for the latter. 
>  
> 
> Would the Krylov solvers be blocking whilst the GPUs are in use running the solvers, or would the host code be able to continue and carry out other calculations whilst waiting for the GPU code to finish? We may need to modify our algorithm to allow for this, but it would make sense to introduce some concurrency so that the CPUs aren’t idling whilst waiting for the GPUs to complete their work.
> We use asynchronous kernel launch and split-phase communication (VecScatterBegin/End). As long as there is no dependency, you can overlap computations on CPU and GPU, or computations with communications. 
>  
> 
> Finally, I’m trying to get the OpenCL PETSc to work on my laptop (Macbook Pro with discrete AMD Radeon R9 M370X GPU). This is mostly because our GPU cluster is out of action until at least late June and I want to get a head start on experimenting with GPUs and TPLS. When I try to run TPLS with the ViennaCL PETSc it reports that my GPU is unable to support double precision. I confirmed that my discrete GPU does support this, however my integrated GPU (Intel Iris) does not. I suspect that ViennaCL is using my integrated GPU instead of my discrete one (it is listed as GPU 0 by OpenCL, with the AMD card is GPU 1). Is there any way of getting PETSc to report which OpenCL device is in use, or to select which device to use? I saw there was some discussion about this on the mailing list archives but I couldn’t find any conclusion.
> No experience. Karl Rupp (cc'ed) might know.
>  
> 
> Thanks in advance for your help,
> 
> Regards,
> 
> Gordon
> 
> -----------------------------------------------
> Dr Gordon P S Gibb
> EPCC, The University of Edinburgh
> Tel: +44 131 651 3459
> 
> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200609/5818ef82/attachment.html>