[petsc-users] Eisenstat-Walker method with GPU assembled matrices

[Jed Brown]

Héctor Barreiro Cabrera <hecbarcab at gmail.com> writes:

> Hello fellow PETSc users,
>
> Following up my previous email
> <https://lists.mcs.anl.gov/pipermail/petsc-users/2020-September/042511.html>,
> I managed to feed the entry data to a SeqAICUSPARSE matrix through a CUDA
> kernel using the new MatCUSPARSEGetDeviceMatWrite function (thanks Barry
> Smith and Mark Adams!). However, I am now facing problems when trying to
> use this matrix within a SNES solver with the Eisenstat-Walker method
> enabled.

Before going further, the error message makes this look like you're using -pc_type eisenstat (a "trick" to reduce the cost of Gauss-Seidel with Krylov) rather than -snes_ksp_ew (the Eisenstat-Walker method for tuning linear solver tolerances within SNES).  Is this what you intend?

> According to PETSc's error log, the preconditioner is failing to invert the
> matrix diagonal. Specifically it says that:
> [0]PETSC ERROR: Arguments are incompatible
> [0]PETSC ERROR: Zero diagonal on row 0
> [0]PETSC ERROR: Configure options PETSC_ARCH=win64_vs2019_release
> --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-clanguage=C++
> --with-fc=0 --with-mpi=0 --with-cuda=1 --with-cudac="win32fe nvcc"
> --with-cuda-dir=~/cuda --download-f2cblaslapack=1 --with-precision=single
> --with-64-bit-indices=0 --with-single-library=1 --with-endian=little
> --with-debugging=0 --with-x=0 --with-windows-graphics=0
> --with-shared-libraries=1 --CUDAOPTFLAGS=-O2
>
> The stack trace leads to the diagonal inversion routine:
> [0]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1913 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c
> [0]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1944 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c
> [0]PETSC ERROR: #3 MatSOR() line 4005 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\INTERF~1\matrix.c
> [0]PETSC ERROR: #4 PCPreSolve_Eisenstat() line 79 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\impls\eisens\eisen.c
> [0]PETSC ERROR: #5 PCPreSolve() line 1549 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\INTERF~1\precon.c
> [0]PETSC ERROR: #6 KSPSolve_Private() line 686 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c
> [0]PETSC ERROR: #7 KSPSolve() line 889 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c
> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 225 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\impls\ls\ls.c
> [0]PETSC ERROR: #9 SNESSolve() line 4567 in
> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\INTERF~1\snes.c
>
> I am 100% positive that the diagonal does not contain a zero entry, so my
> suspicions are either that this operation is not supported on the GPU at
> all (MatInvertDiagonal_SeqAIJ seems to access host-side memory) or that I
> am missing some setting to make this work on the GPU. Is this correct?
>
> Thanks!
>
> Cheers,
> Héctor