[petsc-users] error with ASM and GPUs

[Randall Mackie]

We are investigating the use of porting our PETSc code to work on GPUS.
We have made some good progress but are encountering and error when specifying ASM as the preconditioner.

Specifically, we set:

-ksp_type bcgs
-pc_type asm
-sub_pc_type ilu

That bombs out with the message below.


  Residual norms for em_ solve.
  0 KSP preconditioned resid norm 9.464272188534e+06 true resid norm 2.587899098807e+02 ||r(i)||/||b|| 1.000000000000e+00
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
[0]PETSC ERROR: /home/everderio/DEV/bin/mtd3fwd_real_debug on a linux-intel-intelmpi-real-debug-cuda_2018 named GPU by root Mon Feb 18 16:46:14 2019
[0]PETSC ERROR: Configure options --with-clean=1 --with-debugging=1 --with-fortran=1 --with-blaslapack-dir=/opt/intel/mkl --with-cuda=1 --with-cudac=/usr/local/cuda-10.0/bin/nvcc
[0]PETSC ERROR: #1 VecCUDAGetArrayReadWrite() line 1210 in /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
[0]PETSC ERROR: #2 VecScatterCUDA_StoS() line 272 in /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/vecscattercuda.cu
[0]PETSC ERROR: #3 VecScatterBegin_SGToSS_Stride1() line 384 in /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/impls/vscat.c
[0]PETSC ERROR: #4 VecScatterBegin() line 110 in /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/interface/vscatfce.c
[0]PETSC ERROR: #5 PCApply_ASM() line 486 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/impls/asm/asm.c
[0]PETSC ERROR: #6 PCApply() line 462 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #7 PCApplyBAorAB() line 691 in /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #8 KSP_PCApplyBAorAB() line 309 in /home/everderio/DEV/petsc-3.10.3/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #9 KSPSolve_BCGS() line 87 in /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
[0]PETSC ERROR: #10 KSPSolve() line 780 in /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
 

However, using the following options work:

-ksp_type bcgs
-pc_type bjacobi
-sub_pc_type ilu


Thanks, Randy M.