<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Junchao,</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We have recently been using ASM + LU for 2D problems on both CPU and GPU. However, I found that this method has very bad weak scaling. I find that the cost of PCApply increases by about a factor of 4 each time I increase the problem size in 1 dimension by a
factor of 2 while keeping the load per core/gpu the same. The total number of GMRES iterations does not increase, just the cost of PCApply (and PCSetup). Is this scaling behavior expected? Any ideas of how to optimize the preconditioner?</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you.</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Justin</div>
<div dir="ltr" style="font-family: Aptos, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="mail-editor-reference-message-container">
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"></div>
<div class="ms-outlook-mobile-reference-message skipProofing" style="text-align: left; padding: 3pt 0in 0in; border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) currentcolor currentcolor; font-family: Aptos; font-size: 12pt; color: black;">
<b>From: </b>Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Date: </b>Monday, April 14, 2025 at 7:35 PM<br>
<b>To: </b>Angus, Justin Ray <angus1@llnl.gov><br>
<b>Cc: </b>petsc-dev@mcs.anl.gov <petsc-dev@mcs.anl.gov>, Ghosh, Debojyoti <ghosh5@llnl.gov><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<br>
<br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse).</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing">You can try to see if it works for your case.</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="gmail_signature">--Junchao Zhang</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="ms-outlook-mobile-reference-message skipProofing"><br>
</div>
<div dir="ltr" class="gmail_attr">On Mon, Apr 14, 2025 at 4:39 PM Angus, Justin Ray via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>> wrote:</div>
<blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204);">
<p class="MsoNormal"><span style="font-size: 11pt;">Hello,</span></p>
<p class="MsoNormal"><span style="font-size: 11pt;"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt;">A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using a CPU-based code. I found online where it is stated that
“Parts of most preconditioners run directly on the GPU” (<a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!bw6qeKcY7MKSvlEgcogdKR7fpjZSOFvka6zfDprUZ_sJHdE-YZmRD6UTqWQW3_uGVBII4P-AG0zaGTLbI67_fQ$" target="_blank" style="margin-top: 0px; margin-bottom: 0px;">https://petsc.org/release/faq/</a>).
Is ASM + ILU also available for GPU platforms?</span></p>
<p class="MsoNormal"><span style="font-size: 11pt;"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt;">-Justin</span></p>
</blockquote>
</div>
</body>
</html>