[petsc-dev] HYPRE setup for GPU and CPU solves and other GPU solver questions

[Andrew Ho]

Hi,

I noticed that when I compile PETSc/HYPRE with GPU support, it demands that
I use GPU vectors/matrices (in the form of either
VECCUDA/MATMPIAIJCUSPARSE,  VECHIP/MATMPIAIJHIPSPARSE, or
VECKOKKOS/MATMPIAIJKOKKOS). However, I would like to do some
comparisons/tuning vs. CPU solvers for my particular application, and when
I try to pass in regular CPU vectors/sparse matrices, PETSc complains:

[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to
enable PETSc device support, for example, in some cases, -vec_type cuda
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.19.1-126-g02e62876438
 GIT Date: 2023-04-30 09:10:41 -0500
[0]PETSC ERROR: ./myprog on a  named butter by pb Sun Apr 30 23:07:05 2023
[0]PETSC ERROR: Configure options --prefix=/home/pb/opt/petsc/3.19.1
--with-debugging=0 --with-large-file-io=1 --with-hdf5-dir=/home/pb/.local
--download-hypre --download-metis --download-parmetis
--download-superlu_dist --COPTFLAGS=-O3 --FOPTFLAGS=-O3 --CXXOPTFLAGS=-O3
--CUDAOPTFLAGS=-O3 --with-64-bit-indices=1 --download-slepc
--download-fblaslapack --with-cuda --with-kokkos-dir=/home/pb/.local
--with-kokkos-kernels-dir=/home/pb/.local --with-openmp
[0]PETSC ERROR: #1 VecGetArrayForHYPRE() at
/scratch/pb/code/third_party/petsc/petsc/src/vec/vec/impls/hypre/vhyp.c:95
[0]PETSC ERROR: #2 VecHYPRE_IJVectorPushVecRead() at
/scratch/pb/code/third_party/petsc/petsc/src/vec/vec/impls/hypre/vhyp.c:138
[0]PETSC ERROR: #3 PCApply_HYPRE() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/pc/impls/hypre/hypre.c:433
[0]PETSC ERROR: #4 PCApply() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/pc/interface/precon.c:441

[0]PETSC ERROR: #5 KSP_PCApply() at
/scratch/pb/code/third_party/petsc/petsc/include/petsc/private/kspimpl.h:381
[0]PETSC ERROR: #6 KSPInitialResidual() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/ksp/interface/itres.c:64
[0]PETSC ERROR: #7 KSPSolve_GMRES() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/ksp/impls/gmres/gmres.c:226
[0]PETSC ERROR: #8 KSPSolve_Private() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/ksp/interface/itfunc.c:898
[0]PETSC ERROR: #9 KSPSolve() at
/scratch/pb/code/third_party/petsc/petsc/src/ksp/ksp/interface/itfunc.c:1070
[0]PETSC ERROR: #10 main() at
/home/pb/code/misc/kokkos_test/src/main.cpp:257
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -pc_hypre_type boomeramg (source: command line)
[0]PETSC ERROR: -pc_type hypre (source: command line)
[0]PETSC ERROR: -use_gpu_aware_mpi 0 (source: command line)

Do I have to use a separate PETSc/HYPRE build to do the CPU/GPU comparisons?

I was also wondering if the base PETSc KSP solvers use any kind of GPU
acceleration, and if this is configurable at runtime.

I tried solving a 2D poisson system, and whether I allocated CPU or GPU
vectors the KSP solver will complete in roughly the same non-trivial amount
of time, which makes me suspect that either it is never being solved on the
GPU or is always solved on the GPU since I'm only using a single CPU core.
This is with PETSC compiled with CUDA/Kokkos support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20230430/6dc46da8/attachment.html>