[petsc-users] Performance with GPU and multiple MPI processes per GPU

[Barry Smith]

 Let me try to understand your setup. 

You have two physical GPUs and a CPU with at least 16 physical cores? 

You run with 16 MPI processes, each using its own "virtual" GPU (via MPS). Thus, a single physical GPU is shared by 8 MPI processes?

What happens if you run with 4 MPI processes, compared with 2? 

Can you run with -log_view and send the output when using 2, 4, and 8 MPI processes?  

Barry


> On Jan 19, 2026, at 5:52 AM, Gabriele Penazzi via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi.
> 
> I am using PETSc conjugate gradient liner solver with GPU acceleration (CUDA), on multiple GPUs and multiple MPI processes.
> 
> I noticed that the performances degrade significantly when using multiple MPI processes per GPU, compared to using a single process per GPU.
> For example, 2 GPUs with 2 MPI processes will be about 40% faster than running the same calculation with 2 GPUs and 16 MPI processes.
> 
> I would assume the natural MPI/GPU affinity would be 1-1, however the rest of my application can benefit from multiple MPI processes driving GPU via nvidia MPS, therefore I am trying to understand if this is expected, if I am possibly missing something in the initialization/setup, or if my best choice is to constrain 1-1 MPI/GPU access especially for the PETSc linear solver step. I could not find explicit information about it in the manual.
> 
> Is there any user or maintainer who can tell me more about this use case?
>  
> Best Regards,
> Gabriele Penazzi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260120/d5ef54b4/attachment.html>