<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1, li.gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1, div.gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1
{mso-style-name:gmail-m_-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1
{mso-style-name:gmail-m_-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1;}
span.gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space
{mso-style-name:gmail-m_-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I tried to solve a 3D Poisson (ksp/tutorial/ex45) using -pc_type hypre on gpus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">./ex45 -da_grid_x 128 -da_grid_y 128 -da_grid_z 128 -dm_mat_type hypre -dm_vec_type cuda -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor -log_view<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I profiled the run using NSYS - attached you find all the relevant files.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Looking at the profile I doubt if hypre is running on gpus. The cuda kernels are barely active.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I don’t see any cuda kernel relevant to solve.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Is my assessment correct? How can I verify if hypre indeed is running on gpus?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Karthik.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Mark Adams <mfadams@lbl.gov><br>
<b>Date: </b>Friday, 8 October 2021 at 18:47<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam@stfc.ac.uk><br>
<b>Cc: </b>"petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think you would want to use 'cuda' vec_type, but I .<o:p></o:p></p>
<div>
<p class="MsoNormal">You might ask Hypre how one verifies that the GPU is used.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Mark<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Oct 8, 2021 at 1:35 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Yes, I used it for both cpu and gpu. Is that not okay?</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif">For gpu:
<span style="color:black">-dm_mat_type hypre -dm_vec_type mpicuda </span></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif;color:black"> </span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif;color:black">For cpu: -dm_mat_type hypre -dm_vec_type mpi</span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12.0pt">mfadams@lbl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 18:28<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12.0pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Did you use -dm_mat_type hypre on the GPU case ?</p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Oct 8, 2021 at 12:19 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I tried a different exercise ran the same problem on two cpu cores and on two gpu:</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On gpu
</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">PCApply 6 1.0 6.0335e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00 15 0 0 0 1 15 0 0 0 1 0 0 0 0.00e+00 5 9.65e+01 0</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">and on cpu</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">PCApply 6 1.0 5.6348e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 0</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">timings again are close but gpu version did a reduction 6.0e+00 but the cpu version did not 0.0e+00.
</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I am not sure if that is any indication if hypre ran on gpus?</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Karthik.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12.0pt">mfadams@lbl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 16:36<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12.0pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Oct 8, 2021 at 10:29 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The PCApply timing on</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">gpu</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1">
<span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">PCApply</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">6 1.0 1.0235e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 39</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">39</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0 0.00e+00</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0 0.00e+00</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span></p>
<p class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1">
<span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1"> </span></p>
<p class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1">
<span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">and cpu
</span></p>
<p class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1">
</p>
<p class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803p1">
<span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">PCApply</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">6 1.0 1.0242e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 41</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">41</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803apple-converted-space">
</span><span class="gmail-m-6278275758514210679gmail-m7139920988130781738gmail-m-5205965745375972803s1">0</span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">You don't have GPUs. probably.</p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Use <span style="font-family:Menlo;color:black">-dm_mat_type hypre</span>. </p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">are close. It is hard for me tell if hypre on gpu is on or not.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best,</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Karthik.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12.0pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 14:55<br>
<b>To: </b>Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12.0pt">mfadams@lbl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks Mark, I will try your recommendations.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Should I also change
<span style="font-family:Menlo;color:black">-dm_vec_type to hypre </span>currently I have it as mpicuda?</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Karthik.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Mark Adams <</span><a href="mailto:mfadams@lbl.gov" target="_blank"><span style="font-size:12.0pt">mfadams@lbl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Friday, 8 October 2021 at 14:33<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <</span><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12.0pt">karthikeyan.chockalingam@stfc.ac.uk</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Cc: </b>"</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">" <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12.0pt">petsc-users@mcs.anl.gov</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] hypre on gpus</span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hypre does not record its flops with PETSc's timers.</p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Configure with and without CUDA and see if the timings change in PCApply.</p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hypre does not dynamically switch between CUDA and CPU solves at this time, but you want to use <span style="font-family:Menlo;color:black">-dm_mat_type hypre.</span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:Menlo;color:black">Mark</span></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Oct 8, 2021 at 6:59 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:</p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello,</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I am trying to run ex45 (in KSP tutorial) using hypre on gpus. I have attached the python configuration file and -log_view output from running the below command options</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:Menlo;color:black">mpirun -n 2 ./ex45 -log_view -da_grid_x 169 -da_grid_y 169 -da_grid_z 169 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type
gmres -pc_type hypre -pc_hypre_type boomeramg -ksp_gmres_restart 31 -pc_hypre_boomeramg_strong_threshold 0.7 -ksp_monitor</span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:Menlo;color:black"> </span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The problem was solved and converged but from the output file I suspect hypre is not running on gpus as PCApply and DMCreate does
<b>not</b> record any gpu Mflop/s. However, some events such KSPSolve, MatMult etc are running on gpus.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can you please let me know if I need to add any extra flag to the attached arch-ci-linux-cuda11-double-xx.py script file to get hypre working on gpus?</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Karthik.</p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:Menlo;color:#C00000"> </span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> </p>
</div>
<p><span style="font-size:6.0pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>