<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class=""><br class=""></div> Stefano has mentioned this before. He reported cuSparse matrix-matrix vector products use a very amount of memory.<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 6, 2021, at 4:33 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">I am running out of memory in GAMG. It looks like this is from the new cuSparse RAP.<div class="">I was able to run Hypre with twice as much work on the GPU as this run.</div><div class="">Are there parameters to tweek for this perhaps or can I disable it?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Mark <br class=""><div class=""><br class=""></div><div class=""> 0 SNES Function norm 5.442539952302e-04 <br class="">[2]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br class="">[2]PETSC ERROR: GPU resources unavailable <br class="">[2]PETSC ERROR: CUDA error 2 (cudaErrorMemoryAllocation) : out of memory. Reports alloc failed; this indicates the GPU has run out resources<br class="">[2]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class="">[2]PETSC ERROR: Petsc Development GIT revision: v3.15.1-569-g270a066c1e GIT Date: 2021-07-06 03:22:54 -0700<br class="">[2]PETSC ERROR: ../ex2 on a arch-cori-gpu-opt-gcc named cgpu11 by madams Tue Jul 6 13:37:43 2021<br class="">[2]PETSC ERROR: Configure options --with-mpi-dir=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc --with-cuda-dir=/usr/common/software/sles15_cgpu/cuda/11.1.1 --CFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECI<br class="">ES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --FFLAGS=" -g " -<br class="">-COPTFLAGS=" -O3" --CXXOPTFLAGS=" -O3" --FOPTFLAGS=" -O3" --download-fblaslapack=1 --with-debugging=0 --with-mpiexec="srun -G 1" --with-cuda-gencodearch=70 --with-batch=0 --with-cuda=1 --download-p4est=1 --<br class="">download-hypre=1 --with-zlib=1 PETSC_ARCH=arch-cori-gpu-opt-gcc<br class="">[2]PETSC ERROR: #1 MatProductSymbolic_SeqAIJCUSPARSE_SeqAIJCUSPARSE() at /global/u2/m/madams/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu:2622/" class="">aijcusparse.cu:2622</a><br class="">[2]PETSC ERROR: #2 MatProductSymbolic_ABC_Basic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:1159<br class="">[2]PETSC ERROR: #3 MatProductSymbolic() at /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:799<br class="">[2]PETSC ERROR: #4 MatPtAP() at /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9626<br class="">[2]PETSC ERROR: #5 PCGAMGCreateLevel_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:87<br class="">[2]PETSC ERROR: #6 PCSetUp_GAMG() at /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:663<br class="">[2]PETSC ERROR: #7 PCSetUp() at /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:1014<br class="">[2]PETSC ERROR: #8 KSPSetUp() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406<br class="">[2]PETSC ERROR: #9 KSPSolve_Private() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:850<br class="">[2]PETSC ERROR: #10 KSPSolve() at /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1084<br class="">[2]PETSC ERROR: #11 SNESSolve_NEWTONLS() at /global/u2/m/madams/petsc/src/snes/impls/ls/ls.c:225<br class="">[2]PETSC ERROR: #12 SNESSolve() at /global/u2/m/madams/petsc/src/snes/interface/snes.c:4769<br class="">[2]PETSC ERROR: #13 TSTheta_SNESSolve() at /global/u2/m/madams/petsc/src/ts/impls/implicit/theta/theta.c:185<br class="">[2]PETSC ERROR: #14 TSStep_Theta() at /global/u2/m/madams/petsc/src/ts/impls/implicit/theta/theta.c:223<br class="">[2]PETSC ERROR: #15 TSStep() at /global/u2/m/madams/petsc/src/ts/interface/ts.c:3571<br class="">[2]PETSC ERROR: #16 TSSolve() at /global/u2/m/madams/petsc/src/ts/interface/ts.c:3968<br class="">[2]PETSC ERROR: #17 main() at ex2.c:699<br class=""></div></div></div>
</div></blockquote></div><br class=""></body></html>