<div dir="ltr"><div dir="ltr">On Fri, Dec 3, 2021 at 7:57 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="gmail-m_-3759899747134481842WordSection1">
<p class="MsoNormal">Hello,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am able to successfully run hypre on gpus but the problem seems to consumption a lot of memory. I ran ksp/ksp/tutorial/ex45 on a grid of 320 x 320 x 320 using 6 gpus by the following options<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="gmail-m_-3759899747134481842p1"><span class="gmail-m_-3759899747134481842s1"><span style="font-size:10pt">mpirun -n 6 ./ex45 -da_grid_x 320 -da_grid_y 320 -da_grid_z 320 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg</span></span><span class="gmail-m_-3759899747134481842apple-converted-space"><span style="font-size:10pt">
</span></span><span class="gmail-m_-3759899747134481842s1"><span style="font-size:10pt">-ksp_monitor -log_view -malloc_dump</span></span><span class="gmail-m_-3759899747134481842apple-converted-space"><span style="font-size:10pt">
</span></span><span class="gmail-m_-3759899747134481842s1"><span style="font-size:10pt">-memory_view -malloc_log<u></u><u></u></span></span></p>
<p class="gmail-m_-3759899747134481842p1"><span class="gmail-m_-3759899747134481842s1"><span style="font-size:10pt"><u></u> <u></u></span></span></p>
<p class="gmail-m_-3759899747134481842p1"><span class="gmail-m_-3759899747134481842s1"><span style="font-size:10pt">From the log_view out (also attached) I get the following memory consumption:</span></span><span style="font-size:10pt"><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Summary of Memory Usage in PETSc<u></u><u></u></p>
<p class="MsoNormal">Maximum (over computational time) process memory: total 9.7412e+09 max 1.6999e+09 min 1.5368e+09<u></u><u></u></p>
<p class="MsoNormal">Current process memory: total 8.1640e+09 max 1.4359e+09 min 1.2733e+09<u></u><u></u></p>
<p class="MsoNormal">Maximum (over computational time) space PetscMalloc()ed: total 7.7661e+08 max 1.3401e+08 min 1.1148e+08<u></u><u></u></p>
<p class="MsoNormal">Current space PetscMalloc()ed: total 1.8356e+06 max 3.0594e+05 min 3.0594e+05<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Each gpu is a Nvidia Tesla V100 – even using 4 gpus the system runs out of cuda memory alloc for the above problem. From the above listed memory output I believe the problem should be able to run on one gpu. Is the memory usage of hypre
not listed include above?</p></div></div></blockquote><div><br></div><div>Yes, we have no way of knowing how much memory Hypre is using on the GPU.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="gmail-m_-3759899747134481842WordSection1">
<p class="MsoNormal">Best,<u></u><u></u></p>
<p class="MsoNormal">Karthik.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p><span style="font-size:6pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span></p></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>