<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Barry,</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for the explanation, though I’m not sure I understand exactly what it’s doing. I at least understand now that the overlap doesn’t simply include degrees of freedom from neighboring cells within the range of the specified overlap for each side and direction
(which is how chat GPT told me it worked).</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For this problem ASM with overlap 16 + LU for the sub pc gave optimal performance. A lower value for the overlap results in too many GMRES iterations. I’m actually surprised that such a large overlap was needed, because the code I’m using is nearly identical
to another and in that code the GMRES iterations stay low with an overlap of 4. Perhaps there is something different in the setup that is not clear.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The PETSc preconditioners seem very powerful, bu it is difficult and time-consuming to find the optimal setup for a given problem given my nascent understanding of how the preconditoners work. Is there someone from PETSc (perhaps yourself) with advanced knowledge
of the preconditions that I could schedule a zoom/webex meeting with to discuss?</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks!</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Justin</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Barry Smith <bsmith@petsc.dev><br>
<b>Date: </b>Wednesday, November 5, 2025 at 2:48 PM<br>
<b>To: </b>Angus, Justin Ray <angus1@llnl.gov><br>
<b>Cc: </b>MFAdams@LBL.GOV <MFAdams@LBL.GOV>, Matthew Knepley <knepley@gmail.com>, petsc-dev@mcs.anl.gov <petsc-dev@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<br>
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> An overlap of 16 is huge and would rarely if ever, be done in practice. It is not surprising that the subproblems become as large as they do with such a large overlap. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> How the overlap is used. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> while (overlap--) {</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> add to the subproblem all degrees of freedom that are coupled by nonzeros in the matrix to all the current degrees of freedom in the subproblem.</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> }</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div class="ms-outlook-mobile-reference-message skipProofing"> So it is grabbing all the neighbors for 16 rounds of grabbing.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<blockquote>
<div class="ms-outlook-mobile-reference-message skipProofing">On Nov 5, 2025, at 3:05 PM, Angus, Justin Ray via petsc-dev <petsc-dev@mcs.anl.gov> wrote:</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
I think the issue is my overlap is too large. Perhaps I don’t fully understand how the overlap parameter is used. Let me explain my setup below.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
My vector of unknowns is the electric field on a Yee grid in a 2D geometry. I’m using 4x4 grid cells per rank. This gives 4*5 = 20 degrees of freedom for each of the two in-plane components of E, and 5*5 = 25 for the out-of-plane component. The total is 65
degrees of freedom per rank. My global problem size is 224x16 on 224 ranks for one case, and 224x32 on 448 ranks for another. Using ASM overlap 4, I get the following for PC sub blocks on a rank:</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
PC Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
type: lu</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
out-of-place factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Reusing fill from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Reusing reordering from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
tolerance for zero pivot 2.22045e-14</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
matrix ordering: nd</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
factor fill ratio given 5., needed 4.02544</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Factored matrix follows:</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Mat Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
type: seqaij</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
<span style="background-color: rgb(255, 255, 0);">rows=402, cols=402</span></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
package used to perform factorization: petsc</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
total: nonzeros=16295, allocated nonzeros=16295</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
not using I-node routines</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
The above is for a 224x16 size domain in 224 total ranks, but I get the same thing for a 224x32 size domain on 448 ranks, which is what I am expected to get.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
However, if I set the overlap to 16 (which is larger than by box size on a given rank), I get the following</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 16pt;">
<b>224x16 gid on 112 ranks: </b></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
PC Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
type: lu</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
out-of-place factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Reusing fill from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Reusing reordering from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
tolerance for zero pivot 2.22045e-14</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
matrix ordering: nd</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
factor fill ratio given 5., needed 6.52557</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Factored matrix follows:</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
Mat Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
type: seqaij</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
<span style="background-color: rgb(255, 255, 0);">rows=1316, cols=1316</span></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
package used to perform factorization: petsc</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
total: nonzeros=95195, allocated nonzeros=95195</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Menlo; font-size: 18px;">
not using I-node routines</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 16pt;">
<b>224x16 gid on 112 ranks: </b></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
PC Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
type: lu</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
out-of-place factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
Reusing fill from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
Reusing reordering from past factorization</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
tolerance for zero pivot 2.22045e-14</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
matrix ordering: nd</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
factor fill ratio given 5., needed 8.59182</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
Factored matrix follows:</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
Mat Object: (sub_) 1 MPI process</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
type: seqaij</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
<span style="background-color: rgb(255, 255, 0);">rows=2632, cols=2632</span></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
package used to perform factorization: petsc</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
total: nonzeros=250675, allocated nonzeros=250675</div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="line-height: normal; margin: 0px; font-family: Monaco; font-size: 18px;">
not using I-node routines</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
In this case, with an overlap much larger than the box size, the rows/cols per rank go up by a factor of 2 when doubling the problem size at fixed work per rank. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Why is this?</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
How exactly is the overlap parameter used?</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Thank you.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
-Justin</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt;">
<b>From: </b>Angus, Justin Ray <angus1@llnl.gov><br>
<b>Date: </b>Wednesday, November 5, 2025 at 8:17 AM<br>
<b>To: </b>MFAdams@LBL.GOV <MFAdams@LBL.GOV>, Matthew Knepley <knepley@gmail.com><br>
<b>Cc: </b>petsc-dev@mcs.anl.gov <petsc-dev@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<br>
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Thanks for the reply.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
The work per block should be the same for the weak scaling. I know LU is not scalable with respect to the block size.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Perhaps our setup is not doing what we think it is doing. I’ll look into it further.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
-Justin</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt;">
<b>From: </b>Mark Adams <mfadams@lbl.gov><br>
<b>Date: </b>Wednesday, November 5, 2025 at 6:14 AM<br>
<b>To: </b>Matthew Knepley <knepley@gmail.com><br>
<b>Cc: </b>Angus, Justin Ray <angus1@llnl.gov>, petsc-dev@mcs.anl.gov <petsc-dev@mcs.anl.gov><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<br>
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">And we do not have sparse LU on GPUs so that is done on the CPU.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">And I don't know why it would not weak scale well. </div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Your results are consistent with just using one process with one domain, (re Matt) while you double the problem size.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="gmail_attr">On Tue, Nov 4, 2025 at 2:27 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);">
<div dir="ltr" class="gmail_quote">On Tue, Nov 4, 2025 at 1:25 PM Angus, Justin Ray via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);">
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Hi Junchao,</div>
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a
factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner?</div>
</blockquote>
<div dir="ltr" class="gmail_quote"><br>
</div>
<div dir="ltr" class="gmail_quote">The cost of PCApply for ASM is dominated by the cost of process-local block solves. You are using LU for the block solve. (Sparse) LU has cost roughly O(N^2) for the apply (depending on the structure of the matrix). So, if
you double the size of a local block, your runtime should increase by about 4x. Thus LU is not a scalable method.</div>
<div dir="ltr" class="gmail_quote"><br>
</div>
<div dir="ltr" class="gmail_quote"> Thanks,</div>
<div dir="ltr" class="gmail_quote"><br>
</div>
<div dir="ltr" class="gmail_quote"> Matt</div>
<div dir="ltr" class="gmail_quote"> </div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);">
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
Thank you.</div>
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
-Justin</div>
<div dir="ltr" class="gmail_quote" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div id="m_2373961528016449945m_3413271761903497376mail-editor-reference-message-container">
<div style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt;">
<b>From: </b>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Date: </b>Monday, April 14, 2025 at 7:35 PM<br>
<b>To: </b>Angus, Justin Ray <<a href="mailto:angus1@llnl.gov" target="_blank">angus1@llnl.gov</a>><br>
<b>Cc: </b><a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a> <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>, Ghosh, Debojyoti <<a href="mailto:ghosh5@llnl.gov" target="_blank">ghosh5@llnl.gov</a>><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<br>
<br>
</div>
<div dir="ltr">Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse).</div>
<div dir="ltr">We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">You can try to see if it works for your case.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr" class="gmail_signature">--Junchao Zhang</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr" class="gmail_attr">On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);">
<p class="MsoNormal"><span style="font-size: 11pt;">Hello,</span></p>
<div style="font-size: 11pt;"> </div>
<p class="MsoNormal"><span style="font-size: 11pt;">A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that
“Parts of most preconditioners run directly on the GPU” (<a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bw6qeKcY7MKSvlEgcogdKR7fpjZSOFvka6zfDprUZ_sJHdE-YZmRD6UTqWQW3_uGVBII4P-AG0zaGTLbI67_fQ$" target="_blank" style="margin-top: 0px; margin-bottom: 0px;">https://petsc.org/release/faq/</a>).
Is ASM + ILU also available for GPU platforms?</span></p>
<div style="font-size: 11pt;"> </div>
<p class="MsoNormal"><span style="font-size: 11pt;">-Justin</span></p>
</blockquote>
</div>
</blockquote>
<div dir="ltr" class="gmail_quote"><br>
</div>
<div dir="ltr" class="gmail_quote"><br>
</div>
<div dir="ltr" class="gmail_quote">--</div>
<div dir="ltr" class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div dir="ltr" class="gmail_signature"><br>
</div>
<div dir="ltr" class="gmail_signature"><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!dXQeQOf4ckc4MRP64tltlc6e1FJgPXuEuzX8tHsTreO_vIP2Lbge1es994i-WdQTd1zpmNP2R9dbEHfLa0v_$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</blockquote>
</div>
</div>
</blockquote>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
</div>
</body>
</html>