[petsc-users] one question about GPU

[Satish Balay]

On Fri, 6 Aug 2010, Yujie wrote:

> Dear PETSc Developers,
> 
> I just took a look at new faq and got the following information,
> 
> "Can PETSc use GPUs to speedup computations?
> 
> PETSc-dev has some support for running portions of the computation on
> Nvidia GPUs. See installation.html for how to install PETSc to use
> GPUs. PETSc has a Vec class VECCUDA that performs almost all the
> vector operations on the GPU. The Mat class MATCUDA performs
> matrix-vector products on the GPU but does not have matrix assembly on
> the GPU yet. Both of these classes run in parallel with MPI. All KPS
> methods, except KSPIBCGS, run all their vector operations on the GPU
> thus, for example Jacobi preconditioned Krylov methods run completely
> on the GPU. Preconditioners are a problem, we could do with some help
> for these. The example src/snes/examples/tutorials/ex47cu.cu
> demonstates how the  nonlinear function evaluation can be done on the
> GPU."
> 
> I have also checked the codes of PETSc. I found the codes relevant to
> GPU are added in "SEQ" not "MPI".

This is incorrect. I see:
src/vec/vec/impls/seq/seqcuda
src/vec/vec/impls/mpi/mpicuda
src/mat/impls/aij/seq/seqcuda
src/mat/impls/aij/mpi/mpicuda

> Also, only few functions are realized currently for Vec and
> Mat. However, from FAQ, "Both of these classes run in parallel with
> MPI". How to realized it regarding current version? Thanks a lot.

If you manage to get cuda & infiniband going [with mvapich] - you can try:

mpiexec -n 4 ./ex19 -da_vec_type mpicuda -da_mat_type mpiaijcuda \
-pc_type none -dmmg_nlevels 1 -da_grid_x 100 -da_grid_y 100 \
-log_summary -mat_no_inode -preload off -cuda_synchronize

Note: you need current petsc-dev for this.

Also check: [at the very end]
http://www.mcs.anl.gov/petsc/petsc-as/snapshots/petsc-dev/docs/installation.html

Satish