[petsc-dev] Parallel calculation on GPU

Karl Rupp rupp at iue.tuwien.ac.at
Wed Aug 20 05:11:18 CDT 2014

Hi Pierre,

 > I have a cluster with nodes of 2 sockets of 4 cores+1 GPU.
> Is there a way to run a calculation with 4*N MPI tasks where
> my matrix is first built outside PETSc, then to solve the
> linear system using PETSc Mat, Vec, KSP on only N MPI
> tasks to adress efficiently the N GPUs ?

as far as I can tell, this should be possible with a suitable 
subcommunicator. The tricky piece, however, is to select the right MPI 
ranks for this. Note that you generally have no guarantee on how the MPI 
ranks are distributed across the nodes, so be prepared for something 
fairly specific to your MPI installation.

> I am playing with the communicators without success, but I
> am surely confusing things...

To keep matters simple, try to get this scenario working with a purely 
CPU-based solve. Once this works, the switch to GPUs should be just a 
matter of passing the right flags. Have a look at PetscInitialize() here:
which mentions that you need to create the subcommunicator of 

Best regards,

More information about the petsc-dev mailing list