[petsc-dev] Improving and stabilizing GPU support

[Karl Rupp]

Hi Paul,

 >>> * Reduce CUSP dependency: The current elementary operations are
>>> mainly realized via CUSP. With better support via CUSPARSE and
>>> CUBLAS, I'd add a separate 'native' CUDA backend so that we can
>>> provide a full set of vector and sparse matrix operations out of the
>>> default NVIDIA toolchain. We will still keep CUSP for its
>>> preconditioners, yet we no longer depend on it.
> Agreed. In the past, I've suggested a -vec_type cuda (not cusp). All the
> CUSP operations can be done with Thrust algorithms. Since Thrust comes
> default with CUDA, one can have only a CUDA dependency.

Yes, I opt for
  -vec_type cuda
if everything needed is shipped with the CUDA toolkit. I even tend to 
avoid Thrust as much as possible and go with CUBLAS/CUSPARSE because we 
get faster compilation and less compiler warnings this way, but that's 
an implementation detail :-)


>>> * Integrate last bits of txpetscgpu package. I assume Paul will
>>> provide a helping hand here.
> Of course. This will go much faster as much of the hard work is done. Do
> people want support for different matrix formats in the CUSP classes :
> i.e. diagonal, ellpack, hybrid? I think the CUSP preconditioners can be
> derived from matrices stored in non-csr format (although they're likely
> just doing a convert under the hood).

Since people keep asking for fast SpMV, we should provide these other 
formats as well (actually, they are partially provided with your update 
to the CUSPARSE bindings already). The main reason for CUSP is the SA 
preconditioner, for which SpMV performance doesn't really matter.


>>> * Documentation: Add a chapter on GPUs to the manual, particularly on
>>> what to expect and what not to expect. Update documentation on
>>> webpage regarding installation.
> I will help with the manual.

Cheers :-)


>>> * Integration of FEM quadrature from SNES ex52. The CUDA part
>>> requiring code generation is not very elegant, while the OpenCL
>>> approach is better suited for a library integration thanks to JIT.
>>> However, this requires user code to be provided as a string (again
>>> not very elegant) or loaded from file (more reasonable). How much FEM
>>> functionality do we want to provide via PETSc?
> Multi-GPU is a highly pressing need, IMO. Need to figure out how to make
> Block Jacobi and ASM run efficiently.

The tricky part here is to balance processes vs. threads vs. GPUs. If we 
use more than one GPU per process, we will duplicate more and more of 
the current MPI logic over time just to move data between GPUs. However, 
if we just use one GPU per process, we will under-utilize the CPU unless 
we have a good interaction with threadcomm.

Best regards,
Karli