[petsc-dev] PETSc init eats too much CUDA memory

[Barry Smith]

Without log_view it does not load any cuBLAS/cuSolve immediately with -log_view it loads all that stuff at startup. You need to go into the PetscInitialize() routine find where it loads the cublas and cusolve and comment out those lines then run with -log_view


> On Jan 7, 2022, at 11:14 AM, Zhang, Hong via petsc-dev <petsc-dev at mcs.anl.gov> wrote:
> 
> When PETSc is initialized, it takes about 2GB CUDA memory. This is way too much for doing nothing. A test script is attached to reproduce the issue. If I remove the first line "import torch", PETSc consumes about 0.73GB, which is still significant. Does anyone have any idea about this behavior?
> 
> Thanks,
> Hong
> 
> hongzhang at gpu02:/gpfs/jlse-fs0/users/hongzhang/Projects/pnode/examples (caidao22/update-examples)$ python3 test.py
> CUDA memory before PETSc 0.000GB
> CUDA memory after PETSc 0.004GB
> hongzhang at gpu02:/gpfs/jlse-fs0/users/hongzhang/Projects/pnode/examples (caidao22/update-examples)$ python3 test.py -log_view :0.txt
> CUDA memory before PETSc 0.000GB
> CUDA memory after PETSc 1.936GB
> 
> import torch
> import sys
> import os
> 
> import nvidia_smi
> nvidia_smi.nvmlInit()
> handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
> info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
> print('CUDA memory before PETSc %.3fGB' % (info.used/1e9))
> 
> petsc4py_path = os.path.join(os.environ['PETSC_DIR'],os.environ['PETSC_ARCH'],'lib')
> sys.path.append(petsc4py_path)
> import petsc4py
> petsc4py.init(sys.argv)
> handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
> info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
> print('CUDA memory after PETSc %.3fGB' % (info.used/1e9))
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20220107/1a86f3ef/attachment.html>