[petsc-users] Eisenstat-Walker method with GPU assembled matrices

[Héctor Barreiro Cabrera]

Apologies, I realized that minutes after I hit the send button. My
understanding was that both needed to be user together, but from your email
it's clear that's not the case. After testing the solver without the
preconditioner, but with EW enabled within SNES, everything's working as
expected.

Thank you, and sorry for the noise!

El jue., 15 oct. 2020 a las 15:10, Jed Brown (<jed at jedbrown.org>) escribió:

> Héctor Barreiro Cabrera <hecbarcab at gmail.com> writes:
>
> > Hello fellow PETSc users,
> >
> > Following up my previous email
> > <
> https://lists.mcs.anl.gov/pipermail/petsc-users/2020-September/042511.html
> >,
> > I managed to feed the entry data to a SeqAICUSPARSE matrix through a CUDA
> > kernel using the new MatCUSPARSEGetDeviceMatWrite function (thanks Barry
> > Smith and Mark Adams!). However, I am now facing problems when trying to
> > use this matrix within a SNES solver with the Eisenstat-Walker method
> > enabled.
>
> Before going further, the error message makes this look like you're using
> -pc_type eisenstat (a "trick" to reduce the cost of Gauss-Seidel with
> Krylov) rather than -snes_ksp_ew (the Eisenstat-Walker method for tuning
> linear solver tolerances within SNES).  Is this what you intend?
>
> > According to PETSc's error log, the preconditioner is failing to invert
> the
> > matrix diagonal. Specifically it says that:
> > [0]PETSC ERROR: Arguments are incompatible
> > [0]PETSC ERROR: Zero diagonal on row 0
> > [0]PETSC ERROR: Configure options PETSC_ARCH=win64_vs2019_release
> > --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-clanguage=C++
> > --with-fc=0 --with-mpi=0 --with-cuda=1 --with-cudac="win32fe nvcc"
> > --with-cuda-dir=~/cuda --download-f2cblaslapack=1 --with-precision=single
> > --with-64-bit-indices=0 --with-single-library=1 --with-endian=little
> > --with-debugging=0 --with-x=0 --with-windows-graphics=0
> > --with-shared-libraries=1 --CUDAOPTFLAGS=-O2
> >
> > The stack trace leads to the diagonal inversion routine:
> > [0]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1913 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c
> > [0]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1944 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c
> > [0]PETSC ERROR: #3 MatSOR() line 4005 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\INTERF~1\matrix.c
> > [0]PETSC ERROR: #4 PCPreSolve_Eisenstat() line 79 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\impls\eisens\eisen.c
> > [0]PETSC ERROR: #5 PCPreSolve() line 1549 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\INTERF~1\precon.c
> > [0]PETSC ERROR: #6 KSPSolve_Private() line 686 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c
> > [0]PETSC ERROR: #7 KSPSolve() line 889 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c
> > [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 225 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\impls\ls\ls.c
> > [0]PETSC ERROR: #9 SNESSolve() line 4567 in
> > C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\INTERF~1\snes.c
> >
> > I am 100% positive that the diagonal does not contain a zero entry, so my
> > suspicions are either that this operation is not supported on the GPU at
> > all (MatInvertDiagonal_SeqAIJ seems to access host-side memory) or that I
> > am missing some setting to make this work on the GPU. Is this correct?
> >
> > Thanks!
> >
> > Cheers,
> > Héctor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20201015/acca8355/attachment.html>