<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
p.gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph, li.gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph, div.gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph
{mso-style-name:gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space
{mso-style-name:gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:666518445;
mso-list-template-ids:760881782;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word;-webkit-nbsp-mode: space;line-break:after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Barry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks for your comment.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I took max time as 2.868e+02 sec from the below table, as the total time taken to solve the entire problem. Was I correct in my assumption? <o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Using this max time, I manually tried to calculate the individual event percentage to see if it matched up to T%.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> Max Max/Min Avg Total<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Time (sec): 2.868e+02 1.000 2.868e+02<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Objects: 3.800e+01 1.000 3.800e+01<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Flop: 8.659e+11 1.004 8.642e+11 1.728e+12<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Flop/sec: 3.019e+09 1.004 3.013e+09 6.026e+09<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Memory: 1.764e+10 1.004 1.760e+10 3.521e+10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">MPI Messages: 3.430e+02 1.000 3.430e+02 6.860e+02<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">MPI Message Lengths: 7.134e+08 1.000 2.080e+06 1.427e+09<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">MPI Reductions: 4.637e+03 1.000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Karthik.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Barry Smith <bsmith@petsc.dev><br>
<b>Date: </b>Friday, 1 October 2021 at 16:03<br>
<b>To: </b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <karthikeyan.chockalingam@stfc.ac.uk><br>
<b>Cc: </b>"petsc-users@mcs.anl.gov" <petsc-users@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-users] (percent time in this phase)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"> What is "max time"? It is best to gather timings with a stage PetscLogStagePush() to get a separate subtable for exactly the part of the code you want timing for. For example if you are studying linear solver time you want only the solver
part of the code in the stage, not the time to build the matrix and right hand side.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> It is very difficult to get really correct publishable reliable performance numbers when comparing solvers with similar timings on parallel machines and especially GPUs. Values can be very dependent on particular compilers used, the specific
hardware used, generation of memory used etc.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Oct 1, 2021, at 8:51 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi Barry,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Yes, looks like it is computationally faster using GPUs. I used block jacobi as the preconditioner.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I have attached the output data for cpu and gpu using -ksp_view. I am not sure; what information I should be looking at using -ksp_view?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I have an outstanding question,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="419" style="width:314.0pt;border-collapse:collapse">
<tbody>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">event time</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">T%</span><o:p></o:p></p>
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">cal = (event/max)*100</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">max time</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">2.87E+02</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
</td>
</tr>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">KSPSolve</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">1.58E+02</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">53</span><o:p></o:p></p>
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">55.2</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">MatMult </span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">1.08E+01</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">4</span><o:p></o:p></p>
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">3.76</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">PCApply </span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">1.31E+02</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">37</span><o:p></o:p></p>
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">45.6</span><o:p></o:p></p>
</td>
</tr>
<tr style="height:16.0pt">
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">VecNorm</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">6.23E+01</span><o:p></o:p></p>
</td>
<td width="87" nowrap="" valign="bottom" style="width:65.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">11</span><o:p></o:p></p>
</td>
<td width="159" nowrap="" valign="bottom" style="width:119.0pt;padding:0cm 5.4pt 0cm 5.4pt;height:16.0pt">
<p class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt">21.7</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="color:white"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Matt couple of days back helped breakdown KSPSolve (53 %) ~ PCApply (37%) + VecNorm (11%) + MatMul (4%)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">However, when I calculate T% manually using max time, the numbers for PCApply and VecNorm are way off as you can see from the above table.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">As a result, the cumulative sum of event time don’t match up to KSPSolve. Can you please let me know what I might be doing wrong?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I will be performing extensive benchmarking of various preconditioners and comparing their performance on cpus and gpus, so this information is critical.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Many thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Thursday, 30 September 2021 at 15:47<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> The MatSolve is no better on the GPUs then on the CPU; while other parts of the computation seem to speed up nicely. What is the result of -ksp_view ? Are you using ILU(0) as the preconditioner, this will not solve well on the GPU, its
solve is essentially sequential. You won't want to use ILU(0) in this way on GPUs.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Sep 30, 2021, at 9:41 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Based on your feedback from yesterday. I was trying to breakdown KSPSolve.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Please find the attached bar plot. The numbers are not adding up at least for GPUs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Your feedback from yesterday were based on T%.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I plotted the time spend on each event, hoping that the cumulative sum would add up to KSPSolve time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Thursday, 30 September 2021 at 13:52<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 30, 2021 at 8:50 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">When comparing the MatSolve data for<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">GPU<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">MatSolve 341 1.0 1.3009e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 36 34 0 0 0 36 34 0 0 0 4536 4538 1 5.34e+02 0 0.00e+00 100<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">MatCUSPARSSolAnl 2 1.0 3.2338e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and CPU<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">MatSolve 352 1.0 1.3553e+02 1.0 1.02e+11 1.0 0.0e+00 0.0e+00 0.0e+00 35 34 0 0 0 35 34 0 0 0 4489<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the time spent is almost the same for this preconditioner. Look like MatCUSPARSSolAnl is called only<span class="apple-converted-space"> </span><u>twice</u><span class="apple-converted-space"> </span>(since I am running on two cores)<span class="apple-converted-space"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpirun -n 2 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type cg -pc_type bjacobi -ksp_monitor<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">So would it be fair to assume MatCUSPARSSolAnl is<span class="apple-converted-space"> </span><u>not<span class="apple-converted-space"> </span></u>accounted for in MatSolve and it is an exclusive event?<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Looks like that.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Thanks<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">KSPSolve (53%) + PCSetup (16%) + DMCreateMat (23%) + MatCUSPARSSolAnl (9%) ~ 100 %<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 16:29<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 29, 2021 at 10:18 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">Thank you!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Just to summarize<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">KSPSolve (53%) + PCSetup (16%) + DMCreateMat (23%) + MatCUSPARSSolAnl (9%) ~ 100 %<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">You didn’t happen to mention how MatCUSPARSSolAnl is accounted for? Am I right in accounting for it as above?<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I am not sure.I thought it might be the GPU part of MatSolve(). I will have to look in the code. I am not as familiar with the GPU part.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">MatCUSPARSSolAnl 2 1.0 3.2338e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 9 0 0 0 0 9 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Finally, I believe the vector events, VecNorn, VecTDot, VecAXPY, and VecAYPX are mutually exclusive?<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Yes.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 11:58<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 29, 2021 at 6:24 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">Thank you Mathew. Now, it is all making sense to me.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">From data file ksp_ex45_N511_gpu_2.txt<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">KSPSolve (53%) + KSPSetup (0%) = PCSetup (16%) + PCApply (37%).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">However, you said “So an iteration would mostly consist of MatMult + PCApply, with some vector work”<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">1) You do one solve, but 2 KSPSetUp()s. You must be running on more than one process and using Block-Jacobi . Half the time is spent in the solve (53%)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<pre style="white-space:pre-wrap"><span style="font-size:10.5pt">KSPSetUp 2 1.0 5.3149e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 1 0 0 0 0 1 0 0 0 0.00e+00 0 0.00e+00 0</span><o:p></o:p></pre>
<pre><span style="font-size:10.5pt">KSPSolve 1 1.0 1.5837e+02 1.1 8.63e+11 1.0 6.8e+02 2.1e+06 4.4e+03 53100100100 95 53100100100 96 10881 11730 1022 6.40e+03 1021 8.17e-03 100</span><o:p></o:p></pre>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2) The preconditioner look like BJacobi-ILU. The setup time is 16%, which is all setup of the individual blocks, and this is all used by the numerical ILU factorization.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New"">PCSetUp 2 1.0 4.9623e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 16 0 0 0 0 16 0 0 0 0 58 0 2 6.93e+03 0 0.00e+00 0 PCSetUpOnBlocks 1 1.0 4.9274e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00
0.0e+00 15 0 0 0 0 15 0 0 0 0 59 0 2 6.93e+03 0 0.00e+00 0</span><o:p></o:p></p>
</div>
<pre><span style="font-size:10.5pt">MatLUFactorNum 1 1.0 4.6126e+01 1.3 1.45e+09 1.0 0.0e+00 0.0e+00 0.0e+00 14 0 0 0 0 14 0 0 0 0 63 0 2 6.93e+03 0 0.00e+00 0</span><o:p></o:p></pre>
<pre><span style="font-size:10.5pt">MatILUFactorSym 1 1.0 2.5110e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0 0.00e+00 0</span><o:p></o:p></pre>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">3) The preconditioner application takes 37% of the time, which is all solving the factors and recorded in MatSolve(). Matrix multiplication takes 4%.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Courier New"">PCApply 341 1.0 1.3068e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 37 34 0 0 0 37 34 0 0 0 4516 4523 1 5.34e+02 0 0.00e+00 100</span><o:p></o:p></p>
</div>
</div>
<pre style="white-space:pre-wrap"><span style="font-size:10.5pt">MatSolve 341 1.0 1.3009e+02 1.6 2.96e+11 1.0 0.0e+00 0.0e+00 0.0e+00 36 34 0 0 0 36 34 0 0 0 4536 4538 1 5.34e+02 0 0.00e+00 100</span><o:p></o:p></pre>
<pre style="white-space:pre-wrap"><span style="font-size:10.5pt">MatMult 341 1.0 1.0774e+01 1.1 2.96e+11 1.0 6.9e+02 2.1e+06 2.0e+00 4 34100100 0 4 34100100 0 54801 66441 2 5.86e+03 0 0.00e+00 100</span><o:p></o:p></pre>
<pre><span style="font-size:10.5pt"> </span><o:p></o:p></pre>
<div>
<p class="MsoNormal">4) The significant vector time is all in norms (11%) since they are really slow on the GPU.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
<pre><span style="font-size:10.5pt">VecNorm 342 1.0 6.2261e+01129.9 4.57e+10 1.0 0.0e+00 0.0e+00 6.8e+02 11 5 0 0 15 11 5 0 0 15 1466 196884 0 0.00e+00 342 2.74e-03 100</span><o:p></o:p></pre>
<pre style="white-space:pre-wrap"><span style="font-size:10.5pt">VecTDot 680 1.0 1.7107e+00 1.3 9.09e+10 1.0 0.0e+00 0.0e+00 1.4e+03 1 10 0 0 29 1 10 0 0 29 106079 133922 0 0.00e+00 680 5.44e-03 100</span><o:p></o:p></pre>
<pre><span style="font-size:10.5pt">VecAXPY 681 1.0 3.2036e+00 1.7 9.10e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 11 0 0 0 1 11 0 0 0 56728 58367 682 5.34e+02 0 0.00e+00 100</span><o:p></o:p></pre>
<pre><span style="font-size:10.5pt">VecAYPX 339 1.0 2.6502e+00 1.8 4.53e+10 1.0 0.0e+00 0.0e+00 0.0e+00 1 5 0 0 0 1 5 0 0 0 34136 34153 339 2.71e-03 0 0.00e+00 100</span><o:p></o:p></pre>
<pre style="white-space:pre-wrap"><span style="font-size:10.5pt"> </span><o:p></o:p></pre>
<div>
<p class="MsoNormal">So the solve time is:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> 53% ~ 37% + 4% + 11%<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">and the setup time is about 16%. I was wrong about the SetUp time being included, as it is outside the event:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <a href="https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/interface/itfunc.c#L852" target="_blank">https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/interface/itfunc.c#L852</a><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">It looks like the remainder of the time (23%) is spent preallocating the matrix.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">The MalMult event is 4 %. How does this event figure into the above equation; if preconditioning (MatMult + PCApply) is included in KSPSolve?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Wednesday, 29 September 2021 at 10:58<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>, "<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 29, 2021 at 5:52 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">That was helpful. I would like to provide some additional details of my run on cpus and gpus. Please find the following attachments:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<ol start="1" type="1">
<li class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="mso-list:l0 level1 lfo1">
graph.pdf a plot showing overall time and various petsc events.<o:p></o:p></li><li class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="mso-list:l0 level1 lfo1">
ksp_ex45_N511_cpu_6.txt data file of the log_summary<span class="apple-converted-space"> </span><o:p></o:p></li><li class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232msolistparagraph" style="mso-list:l0 level1 lfo1">
ksp_ex45_N511_gpu_2.txt data file of the log_summary<span class="apple-converted-space"> </span><o:p></o:p></li></ol>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I used the following petsc options for cpu<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpirun -n 6 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaij -dm_vec_type mpi -ksp_type cg -pc_type bjacobi -ksp_monitor<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and for gpus<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpirun -n 1 ./ex45 -log_summary -da_grid_x 511 -da_grid_y 511 -da_grid_z 511 -dm_mat_type mpiaijcusparse -dm_vec_type mpicuda -ksp_type cg -pc_type bjacobi -ksp_monitor<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to run the following problem<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"><a href="https://petsc.org/release/src/ksp/ksp/tutorials/ex45.c.html" target="_blank">https://petsc.org/release/src/ksp/ksp/tutorials/ex45.c.html</a></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">From the above code, I see is there no individual function called<span class="apple-converted-space"> </span></span>KSPSetUp(), so I gather KSPSetDM, KSPSetComputeInitialGuess, KSPSetComputeRHS, kSPSetComputeOperators
all are timed together as KSPSetUp.<span class="apple-converted-space"><span style="font-size:12.0pt"> </span></span><span style="font-size:12.0pt">For this example, is KSPSetUp time and KSPSolve time mutually exclusive?</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">No, KSPSetUp() will be contained in KSPSolve() if it is called automatically.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">In your response you said that</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> “PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it depends on how much of the preconditioner construction can take place early, so depends exactly on the preconditioner used.”<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I don’t see a explicit call to PCSetUp() or PCApply() in ex45; so for this particular preconditioner (bjacobi) how can I tell how they are timed?</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">They are all inside KSPSolve(). If you have a preconditioned linear solve, the oreconditioning happens during the iteration. So an iteration would mostly<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">consist of MatMult + PCApply, with some vector work.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I am hoping to time KSP solving and preconditioning mutually exclusively.</span><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I am not sure that concept makes sense here. See above.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Matt<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Kind regards,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Karthik.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br>
<b>Date:<span class="apple-converted-space"> </span></b>Tuesday, 28 September 2021 at 19:19<br>
<b>To:<span class="apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Sep 28, 2021, at 12:11 PM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Thanks for Barry for your response.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I was just benchmarking the problem with various preconditioner on cpu and gpu. I understand, it is not possible to get mutually exclusive timing.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">However, can you tell if KSPSolve time includes both PCSetup and PCApply? And if KSPSolve and KSPSetup are mutually exclusive? Likewise for PCSetUp and PCApply.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> If you do not call KSPSetUp() separately from KSPSolve() then its time is included with KSPSolve().<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> PCSetUp() time may be in KSPSetUp() or it maybe in PCApply() it depends on how much of the preconditioner construction can take place early, so depends exactly on the preconditioner used.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> So yes the answer is not totally satisfying. The one thing I would recommend is to not call KSPSetUp() directly and then KSPSolve() will always include the total time of the solve plus all setup time. PCApply will contain all the time
to apply the preconditioner but may also include some setup time.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Best,<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Karthik.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt">From:<span class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></span></b><span style="font-size:12.0pt">Barry
Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br>
<b>Date:<span class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>Tuesday, 28 September 2021 at 16:56<br>
<b>To:<span class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>"Chockalingam, Karthikeyan (STFC,DL,HC)" <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc:<span class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>"<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:<span class="gmail-m-2472759815423112172gmail-m-3492767196240074348gmail-m391186430279648065gmail-m-3409338445930824232apple-converted-space"> </span></b>Re: [petsc-users] %T (percent time in this phase)</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Sep 28, 2021, at 10:55 AM, Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Hello,</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">I ran ex45 in the KPS tutorial, which is a 3D finite-difference Poisson problem. I noticed from the output from using the flag -log_summary that for various events their respective %T (percent time in this
phase) do not add up to 100 but rather exceeds 100. So, I gather there is some overlap among these events. I am primarily looking at the events KSPSetUp, KSPSolve, PCSetUp and PCSolve. Is it possible to get a mutually exclusive %T or Time for these individual
events? I have attached the log_summary output file from my run for your reference.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt"> </span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> For nested solvers it is tricky to get the times to be mutually exclusive because some parts of the building of the preconditioner is for some preconditioners delayed until the solve has started. <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> It looks like you are using the default preconditioner options which for this example are taking more or less no time since so many iterations are needed. It is best to use -pc_type mg to use geometric multigrid on this problem.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> Barry<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Thanks!</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt">Karthik.</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:6.0pt;font-family:Helvetica">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email
or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware
but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><ksp_ex45_N511_cpu_6.txt><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">--<span class="apple-converted-space"> </span><o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><KSPSolve.pdf><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><ksp_ex45_N511_cpu_6.txt><ksp_ex45_N511_gpu_2.txt><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>