[petsc-users] Hypre (GPU) memory consumption

[Karthikeyan Chockalingam - STFC UKRI]

Hello,

I am able to successfully run hypre on gpus but the problem seems to consumption a lot of memory.  I ran ksp/ksp/tutorial/ex45 on a grid of 320 x 320 x 320 using 6 gpus by the following options


mpirun -n 6 ./ex45 -da_grid_x 320 -da_grid_y 320 -da_grid_z 320 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg  -ksp_monitor -log_view -malloc_dump  -memory_view -malloc_log



From the log_view out (also attached) I get the following memory consumption:

Summary of Memory Usage in PETSc
Maximum (over computational time) process memory:           total 9.7412e+09 max 1.6999e+09 min 1.5368e+09
Current process memory:                                                                total 8.1640e+09 max 1.4359e+09 min 1.2733e+09
Maximum (over computational time) space PetscMalloc()ed: total 7.7661e+08 max 1.3401e+08 min 1.1148e+08
Current space PetscMalloc()ed:                                                      total 1.8356e+06 max 3.0594e+05 min 3.0594e+05

Each gpu is a Nvidia Tesla V100 – even using 4 gpus the system runs out of cuda memory alloc for the above problem. From the above listed memory output I believe the problem should be able to run on one gpu. Is the memory usage of hypre not listed include above?

Best,
Karthik.


This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211203/9cfc44e5/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ksp_cg_pc_hypre_ex45_N320_gpu_6.txt
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211203/9cfc44e5/attachment-0001.txt>