<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> We still have the assumption the AIJ matrix always has a copy on the GPU. How did you fill up the matrix on the GPU while not having its copy on the CPU?<div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""> When we remove this assumption we have to add a bunch more code for CPU only things to make sure they properly get the data from the GPU.</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Oct 15, 2020, at 4:16 AM, Héctor Barreiro Cabrera <<a href="mailto:hecbarcab@gmail.com" class="">hecbarcab@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hello fellow PETSc users,<div class=""><br class=""></div><div class="">Following up <a href="https://lists.mcs.anl.gov/pipermail/petsc-users/2020-September/042511.html" class="">my previous email</a>, I managed to feed the entry data to a SeqAICUSPARSE matrix through a CUDA kernel using the new MatCUSPARSEGetDeviceMatWrite function (thanks Barry Smith and Mark Adams!). However, I am now facing problems when trying to use this matrix within a SNES solver with the Eisenstat-Walker method enabled.</div><div class=""><br class=""></div><div class="">According to PETSc's error log, the preconditioner is failing to invert the matrix diagonal. Specifically it says that:</div><div class=""><font face="monospace" class="">[0]PETSC ERROR: Arguments are incompatible<br class="">[0]PETSC ERROR: Zero diagonal on row 0<br class=""></font><div class=""><font face="monospace" class="">[0]PETSC ERROR: Configure options PETSC_ARCH=win64_vs2019_release --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-clanguage=C++ --with-fc=0 --with-mpi=0 --with-cuda=1 --with-cudac="win32fe nvcc" --with-cuda-dir=~/cuda --download-f2cblaslapack=1 --with-precision=single --with-64-bit-indices=0 --with-single-library=1 --with-endian=little --with-debugging=0 --with-x=0 --with-windows-graphics=0 --with-shared-libraries=1 --CUDAOPTFLAGS=-O2</font></div><div class=""></div></div><div class=""><br class=""></div><div class="">The stack trace leads to the diagonal inversion routine:<br class=""><font face="monospace" class="">[0]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1913 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c<br class="">[0]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1944 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c<br class="">[0]PETSC ERROR: #3 MatSOR() line 4005 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\INTERF~1\matrix.c<br class="">[0]PETSC ERROR: #4 PCPreSolve_Eisenstat() line 79 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\impls\eisens\eisen.c<br class="">[0]PETSC ERROR: #5 PCPreSolve() line 1549 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\INTERF~1\precon.c<br class="">[0]PETSC ERROR: #6 KSPSolve_Private() line 686 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c<br class="">[0]PETSC ERROR: #7 KSPSolve() line 889 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c<br class="">[0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 225 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\impls\ls\ls.c<br class="">[0]PETSC ERROR: #9 SNESSolve() line 4567 in C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\INTERF~1\snes.c</font><br class=""></div><div class=""><font face="monospace" class=""><br class=""></font></div>I am 100% positive that the diagonal does not contain a zero entry, so my suspicions are either that this operation is not supported on the GPU at all (MatInvertDiagonal_SeqAIJ seems to access host-side memory) or that I am missing some setting to make this work on the GPU. Is this correct?<div class=""><br class=""></div><div class="">Thanks!</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Héctor</div></div>
</div></blockquote></div><br class=""></div></body></html>