[petsc-users] CUDA running out of memory in PtAP

[Mark Adams]

Can I turn off using cuSprarse for RAP?

On Tue, Jul 6, 2021 at 6:25 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>   Stefano has mentioned this before. He reported cuSparse matrix-matrix
> vector products use a very amount of memory.
>
> On Jul 6, 2021, at 4:33 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
> I am running out of memory in GAMG. It looks like this is from the new
> cuSparse RAP.
> I was able to run Hypre with twice as much work on the GPU as this run.
> Are there parameters to tweek for this perhaps or can I disable it?
>
> Thanks,
> Mark
>
>    0 SNES Function norm 5.442539952302e-04
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: GPU resources unavailable
> [2]PETSC ERROR: CUDA error 2 (cudaErrorMemoryAllocation) : out of memory.
> Reports alloc failed; this indicates the GPU has run out resources
> [2]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [2]PETSC ERROR: Petsc Development GIT revision: v3.15.1-569-g270a066c1e
>  GIT Date: 2021-07-06 03:22:54 -0700
> [2]PETSC ERROR: ../ex2 on a arch-cori-gpu-opt-gcc named cgpu11 by madams
> Tue Jul  6 13:37:43 2021
> [2]PETSC ERROR: Configure options
> --with-mpi-dir=/usr/common/software/sles15_cgpu/openmpi/4.0.3/gcc
> --with-cuda-dir=/usr/common/software/sles15_cgpu/cuda/11.1.1 --CFLAGS="
> -g -DLANDAU_DIM=2 -DLANDAU_MAX_SPECI
> ES=10 -DLANDAU_MAX_Q=4" --CXXFLAGS=" -g -DLANDAU_DIM=2
> -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4" --CUDAFLAGS="-g -Xcompiler
> -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4"
> --FFLAGS="   -g " -
> -COPTFLAGS="   -O3" --CXXOPTFLAGS=" -O3" --FOPTFLAGS="   -O3"
> --download-fblaslapack=1 --with-debugging=0 --with-mpiexec="srun -G 1"
> --with-cuda-gencodearch=70 --with-batch=0 --with-cuda=1 --download-p4est=1
> --
> download-hypre=1 --with-zlib=1 PETSC_ARCH=arch-cori-gpu-opt-gcc
> [2]PETSC ERROR: #1 MatProductSymbolic_SeqAIJCUSPARSE_SeqAIJCUSPARSE() at
> /global/u2/m/madams/petsc/src/mat/impls/aij/seq/seqcusparse/
> aijcusparse.cu:2622
> [2]PETSC ERROR: #2 MatProductSymbolic_ABC_Basic() at
> /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:1159
> [2]PETSC ERROR: #3 MatProductSymbolic() at
> /global/u2/m/madams/petsc/src/mat/interface/matproduct.c:799
> [2]PETSC ERROR: #4 MatPtAP() at
> /global/u2/m/madams/petsc/src/mat/interface/matrix.c:9626
> [2]PETSC ERROR: #5 PCGAMGCreateLevel_GAMG() at
> /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:87
> [2]PETSC ERROR: #6 PCSetUp_GAMG() at
> /global/u2/m/madams/petsc/src/ksp/pc/impls/gamg/gamg.c:663
> [2]PETSC ERROR: #7 PCSetUp() at
> /global/u2/m/madams/petsc/src/ksp/pc/interface/precon.c:1014
> [2]PETSC ERROR: #8 KSPSetUp() at
> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:406
> [2]PETSC ERROR: #9 KSPSolve_Private() at
> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:850
> [2]PETSC ERROR: #10 KSPSolve() at
> /global/u2/m/madams/petsc/src/ksp/ksp/interface/itfunc.c:1084
> [2]PETSC ERROR: #11 SNESSolve_NEWTONLS() at
> /global/u2/m/madams/petsc/src/snes/impls/ls/ls.c:225
> [2]PETSC ERROR: #12 SNESSolve() at
> /global/u2/m/madams/petsc/src/snes/interface/snes.c:4769
> [2]PETSC ERROR: #13 TSTheta_SNESSolve() at
> /global/u2/m/madams/petsc/src/ts/impls/implicit/theta/theta.c:185
> [2]PETSC ERROR: #14 TSStep_Theta() at
> /global/u2/m/madams/petsc/src/ts/impls/implicit/theta/theta.c:223
> [2]PETSC ERROR: #15 TSStep() at
> /global/u2/m/madams/petsc/src/ts/interface/ts.c:3571
> [2]PETSC ERROR: #16 TSSolve() at
> /global/u2/m/madams/petsc/src/ts/interface/ts.c:3968
> [2]PETSC ERROR: #17 main() at ex2.c:699
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20210706/63464c91/attachment-0001.html>