[petsc-dev] Parallel calculation on GPU
rupp at iue.tuwien.ac.at
Wed Aug 20 05:11:18 CDT 2014
> I have a cluster with nodes of 2 sockets of 4 cores+1 GPU.
> Is there a way to run a calculation with 4*N MPI tasks where
> my matrix is first built outside PETSc, then to solve the
> linear system using PETSc Mat, Vec, KSP on only N MPI
> tasks to adress efficiently the N GPUs ?
as far as I can tell, this should be possible with a suitable
subcommunicator. The tricky piece, however, is to select the right MPI
ranks for this. Note that you generally have no guarantee on how the MPI
ranks are distributed across the nodes, so be prepared for something
fairly specific to your MPI installation.
> I am playing with the communicators without success, but I
> am surely confusing things...
To keep matters simple, try to get this scenario working with a purely
CPU-based solve. Once this works, the switch to GPUs should be just a
matter of passing the right flags. Have a look at PetscInitialize() here:
which mentions that you need to create the subcommunicator of
More information about the petsc-dev