[petsc-users] error with ASM and GPUs

[Matthew Knepley]

On Mon, Feb 18, 2019 at 8:43 PM Randall Mackie via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> We are investigating the use of porting our PETSc code to work on GPUS.
> We have made some good progress but are encountering and error when
> specifying ASM as the preconditioner.
>
> Specifically, we set:
>
> -ksp_type bcgs
> -pc_type asm
> -sub_pc_type ilu
>
> That bombs out with the message below.
>

It certainly looks like BJACOBI propagates the matrix type correctly to
submatrices, and ASM does not.
Patrick fixed this for BJacobi I believe. Patrick, do you remember what
ChangeSet fixed it?

  Thanks,

     Matt


>
>   Residual norms for em_ solve.
>   0 KSP preconditioned resid norm 9.464272188534e+06 true resid norm
> 2.587899098807e+02 ||r(i)||/||b|| 1.000000000000e+00
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018
> [0]PETSC ERROR: /home/everderio/DEV/bin/mtd3fwd_real_debug on a
> linux-intel-intelmpi-real-debug-cuda_2018 named GPU by root Mon Feb 18
> 16:46:14 2019
> [0]PETSC ERROR: Configure options --with-clean=1 --with-debugging=1
> --with-fortran=1 --with-blaslapack-dir=/opt/intel/mkl --with-cuda=1
> --with-cudac=/usr/local/cuda-10.0/bin/nvcc
> [0]PETSC ERROR: #1 VecCUDAGetArrayReadWrite() line 1210 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: #2 VecScatterCUDA_StoS() line 272 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/
> vecscattercuda.cu
> [0]PETSC ERROR: #3 VecScatterBegin_SGToSS_Stride1() line 384 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/impls/vscat.c
> [0]PETSC ERROR: #4 VecScatterBegin() line 110 in
> /home/everderio/DEV/petsc-3.10.3/src/vec/vscat/interface/vscatfce.c
> [0]PETSC ERROR: #5 PCApply_ASM() line 486 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/impls/asm/asm.c
> [0]PETSC ERROR: #6 PCApply() line 462 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #7 PCApplyBAorAB() line 691 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #8 KSP_PCApplyBAorAB() line 309 in
> /home/everderio/DEV/petsc-3.10.3/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #9 KSPSolve_BCGS() line 87 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
> [0]PETSC ERROR: #10 KSPSolve() line 780 in
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
>
>
> However, using the following options work:
>
> -ksp_type bcgs
> -pc_type bjacobi
> -sub_pc_type ilu
>
>
> Thanks, Randy M.
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190218/101495b9/attachment.html>