<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.E-postmall19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Dear Barry,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have tried what you suggested on a (not too large) test example on a bigger cluster that I have access to, using -redistribute_ksp_initial_guess_nonzero 0 and
1, respectively. The average timings during the 5 first (design) iterations are 8.7 s (state) and 7.9 s (adjoint) for the case with zero initial guesses and 5.0 s (state) and 5.7 s (adjoint) for the cases with nonzero initial guesses. These solvings are the
bottleneck of my program, accounting for about 60-90% of the total computational time, depending on various parameters. The program is basically consisting of the loop: solve state > solve adjoint > update design > repeat. This is repeated for a couple of
hundred iterations.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">From my experience, the number of iterations to convergence in each state/adjoint solve will decrease when increasing the (design) iterative counter (i.e. the
longer the process has gone on for) IF the initial guess is the solution to the previous solve. This is because the design update is smaller in the end of the process than in the beginning, and a smaller design update leads to smaller changes in state/adjoint
solution between subsequent (design) iterations. This means that the numbers provided above are on the low side: most likely the savings can be even more in the end of the design process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jonas Lundgren<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="SV" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Från:</span></b><span lang="SV" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Barry Smith <bsmith@petsc.dev>
<br>
<b>Skickat:</b> den 21 augusti 2023 20:13<br>
<b>Till:</b> Jonas Lundgren <jonas.lundgren@liu.se><br>
<b>Kopia:</b> petsc-users@mcs.anl.gov<br>
<b>Ämne:</b> Re: [petsc-users] (Sub) KSP initial guess with PCREDISTRIBUTE<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"> When you use 2 KSP so that you can use the previous "solution" as the initial guess for the next problem, how much savings do you get? In iterations inside <span style="font-family:"Calibri",sans-serif">PCREDISTRIBUTE and in time (relative
to the entire linear solver time and relative to the entire application run)? You can get this information running with -log_view</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"> That is, run the 2 KSP simulation twice, once with the inner KSPSetNonzeroInitialGuess() on and once with it off and compare the times for the two cases.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"> Thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"> Barry</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"> Using KSPSetNonzeroInitialGuess() requires an extra matrix-vector product and preconditioner application, so I would like to verify that you have a measurable performance improvement with
the initial guess.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<br>
</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Aug 21, 2023, at 7:06 AM, Jonas Lundgren via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span lang="SV" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Dear PETSc users,</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="SV" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I have a problem regarding the setting of initial guess to KSP when using PCREDISTRIBUTE as the preconditioner. (The reason to why I use PCREDISTRIBUTE is because I have a
lot of fixed DOF in my problem, and PCREDISTRIBUTE successfully reduces the problem size and therefore speeds up the solving).<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">First, some details:<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I
use a version of PETSc 3.19.1<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The
KSP I use is KSPPREONLY, as suggested in the manual pages of PCREDISTRIBUTE:<a href="https://petsc.org/release/manualpages/PC/PCREDISTRIBUTE/"><span style="color:#954F72">https://petsc.org/release/manualpages/PC/PCREDISTRIBUTE/</span></a><o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I
use KSPBCGSL as sub-KSP. I can perfectly well solve my problem using this as my main KSP, but the performance is much worse than when using it as my sub-KSP (under KSPPREONLY+PCREDISTRIBUTE) due to the amount of fixed DOF<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I
am first solving a state problem, then an adjoint problem using the same linear operator.<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The
adjoint vector is used as sensitivity information to update a design. After the design update, the state+adjoint problems are solved again with a slightly updated linear operator. This is done for hundreds of (design) iteration steps<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I
want the initial guess for the state problem to be the state solution from the previous (design) iteration, and same for the adjoint problem<o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">-</span><span style="font-size:7.0pt"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">I
am aware of the default way of setting a custom initial guess: KSPSetInitialGuessNonzero(ksp, PETSC_TRUE) together with providing the actual guess in the x vector in the call to KSPSolve(ksp, b, x)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The main problem is that PCREDISTRIBUTE internally doesn’t use the input solution vector (x) when calling KSPSolve() for the sub-KSP. It zeroes out the solution vector (x)
when starting to build x = diag(A)^{-1} b in the beginning of PCApply_Redistribute(), and uses red->x as the solution vector/initial guess when calling KSPSolve(). Therefore, I cannot reach the sub-KSP with an initial guess.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Additionally, KSPPREONLY prohibits the use of having a nonzero initial guess (the error message says “it doesn’t make sense”). I guess I can remove the line raising this error
and recompile the PETSc libraries, but it still won’t solve the previously mentioned problem, which seems to be the hard nut to crack.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">So far, I have found out that if I create 2 KSP object, one each for the state and adjoint problems, it is enough with calling KSPSetInitialGuessNonzero(subksp, PETSC_TRUE)
on the subksp. It seems as if the variable red->x in PCApply_Redistribute() is kept untouched in memory between calls to the main KSP and therefore is used as (non-zero) initial guess to the sub-KSP. This has been verified by introducing PetscCall(PetscObjectCompose((PetscObject)pc,"redx",(PetscObject)red->x));
in PCApply_Redistribute(), recompiling the PETSc library, and then inserting a corresponding PetscObjectQuery((PetscObject)pc, "redx", (PetscObject *)&redx); in my own program source file.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">However, I would like to only create 1 KSP to be used with both the state and adjoint problems (same linear operator), for memory reasons. When I do this, the initial guesses
are mixed up between the two problems: the initial guess for the adjoint problem is the solution to the state problem in the current design iteration, and the initial guess for the state problem is the solution to the adjoint problem in the previous design
iteration. These are very bad guesses that increases the time to convergence in each state/adjoint solve.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">So, the core of the problem (as far as I can understand) is that I want to control the initial guess red->x in PCApply_Redistribute().<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The only solution I can think of is to include a call to PetscObjectQuery() in PCApply_Redistribute() to obtain a vector with the initial guess from my main program. And then
I need to keep track of the initial guesses for my both problems in my main program myself (minor problem). This is maybe not the neatest way, and I do not know if this approach affects the performance negatively? Maybe one call each to PetscObjectQuery()
and PetscObjectCompose() per call to PCApply_Redistribute() is negligible?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Is there another (and maybe simpler) solution to this problem? Maybe I can SCATTER_FORWARD the input x vector in PCApply_Redistribute() before it is zeroed out, together with
allowing for non-zero initial guess in KSPPREONLY?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Any help is welcome!<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Best regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jonas Lundgren<o:p></o:p></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>