[petsc-dev] PETSc multi-GPU assembly - current status

[Florian Rathgeber]

This is loosely a follow up to [1]. In this thread a few potential ways
for making GPU assembly work with PETSc were discussed and to me the two
most promising appeared to be:
1) Create a PETSc matrix from a pre-assembled CSR structure, or
2) Preallocate a PETSc matrix and get the handle to pass the row
pointer, column indices and values array to a custom assembly routine.

To recap what Lawrence already said in [1]: We're working on PyOP2 [2] a
tool chain for parallel (FEM) computations on unstructured meshes with
support for multiple hardware platforms, currently CPU + MPI + OpenMP
and CUDA. We are using PETSc (MPI)AIJ on the CPU, and thinking how to
make multi-GPU via MPI work, preferably also using PETSc. We compute
local assembly matrices on the GPU and a crucial requirement is that the
matrix *only* lives in device device, we want to avoid any host <->
device data transfers.

So far we have been using CUSP with a custom (generated) assembly into
our own CUSP-compatible CSR data structure for a single GPU. Since CUSP
doesn't give us multi-GPU solvers out of the box we'd rather use
existing infrastructure that works rather than rolling our own.

At the time of [1] supporting GPU assembly in one form or the other was
on the roadmap, but the implementation direction seemed to not have been
finally decided. Was there any progress since then or anything to add to
the discussion? Is there even (experimental) code we might be able to
use? Note that we're using petsc4py to interface to PETSc.

A cursory look through petsc-dev and currently active petsc branches
didn't reveal anything obvious, but apologies if I missed something.

Florian

[1]:
https://lists.mcs.anl.gov/mailman/htdig/petsc-dev/2012-November/009941.html
[2]: https://github.com/OP2/PyOP2

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2913 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130502/ffc9caa1/attachment.p7s>