[petsc-users] Hypre (GPU) memory consumption

[Matthew Knepley]

On Fri, Dec 3, 2021 at 7:57 AM Karthikeyan Chockalingam - STFC UKRI <
karthikeyan.chockalingam at stfc.ac.uk> wrote:

> Hello,
>
>
>
> I am able to successfully run hypre on gpus but the problem seems to
> consumption a lot of memory.  I ran ksp/ksp/tutorial/ex45 on a grid of 320
> x 320 x 320 using 6 gpus by the following options
>
>
>
> mpirun -n 6 ./ex45 -da_grid_x 320 -da_grid_y 320 -da_grid_z 320
> -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre
> -pc_hypre_type boomeramg  -ksp_monitor -log_view -malloc_dump  -memory_view
> -malloc_log
>
>
>
> From the log_view out (also attached) I get the following memory
> consumption:
>
>
>
> Summary of Memory Usage in PETSc
>
> Maximum (over computational time) process memory:           total
> 9.7412e+09 max 1.6999e+09 min 1.5368e+09
>
> Current process memory:
>                               total 8.1640e+09 max 1.4359e+09 min 1.2733e+09
>
> Maximum (over computational time) space PetscMalloc()ed: total 7.7661e+08
> max 1.3401e+08 min 1.1148e+08
>
> Current space PetscMalloc()ed:
>                                     total 1.8356e+06 max 3.0594e+05 min
> 3.0594e+05
>
>
>
> Each gpu is a Nvidia Tesla V100 – even using 4 gpus the system runs out of
> cuda memory alloc for the above problem. From the above listed memory
> output I believe the problem should be able to run on one gpu. Is the
> memory usage of hypre not listed include above?
>

Yes, we have no way of knowing how much memory Hypre is using on the GPU.

  Thanks,

     Matt


> Best,
>
> Karthik.
>
>
>
> This email and any attachments are intended solely for the use of the
> named recipients. If you are not the intended recipient you must not use,
> disclose, copy or distribute this email or any of its attachments and
> should notify the sender immediately and delete this email from your
> system. UK Research and Innovation (UKRI) has taken every reasonable
> precaution to minimise risk of this email or any attachments containing
> viruses or malware but the recipient should carry out its own virus and
> malware checks before opening the attachments. UKRI does not accept any
> liability for any losses or damages which the recipient may sustain due to
> presence of any viruses.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211203/ccafd507/attachment.html>