<div dir="ltr"><div dir="ltr">On Tue, Apr 15, 2025 at 1:45 PM Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com">junchao.zhang@gmail.com</a>> wrote:</div><div class="gmail_quote gmail_quote_container"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, Justin,</div><div> I don't know ASM well enough. I just browsed its code. It seems it has a lot of matrix partitioning and indexing operations, which definitely are not done on GPUs currently.</div><div> But you could still try that, as petsc will copy data from the device to the host as needed to perform host-only operations. You can profile with -log_vew -log_view_gpu_time so that we can see how expensive these operations are.</div><div><br></div><div> Barry should know more about ASM. </div></div></blockquote><div><br></div><div>Usually, ASM is a tactic for solving in parallel. Each process has a diagonal matrix block. ASM means overlapping blocks,</div><div>whereas BJacobi means non-overlapping. The block will likely be extracted on the CPU, but pushed down to the GPU for solves.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>--Junchao Zhang</div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 15, 2025 at 10:43 AM Angus, Justin Ray <<a href="mailto:angus1@llnl.gov" target="_blank">angus1@llnl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11pt">Hi Junchao,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Thanks for the reply.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Does ASM work the same on GPU systems as it does on CPU systems?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div id="m_1444264047744260400m_-4952117823530399171mail-editor-reference-message-container">
<div>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Date: </b>Monday, April 14, 2025 at 7:35</span><span style="font-size:12pt;font-family:Arial,sans-serif;color:black"> </span><span style="font-size:12pt;color:black">PM<br>
<b>To: </b>Angus, Justin Ray <<a href="mailto:angus1@llnl.gov" target="_blank">angus1@llnl.gov</a>><br>
<b>Cc: </b><a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a> <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>, Ghosh, Debojyoti <<a href="mailto:ghosh5@llnl.gov" target="_blank">ghosh5@llnl.gov</a>><br>
<b>Subject: </b>Re: [petsc-dev] Additive Schwarz Method + ILU on GPU platforms<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">Petsc supports ILU0/ICC0 numeric factorization (without reordering) and then triangular solve on GPUs. It is done by calling vendor libraries (ex. cusparse).<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">We have options -pc_factor_mat_factor_on_host <bool> -pc_factor_mat_solve_on_host <bool> to force doing the factorization and MatSolve on the host for device matrix types.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">You can try to see if it works for your case.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">--Junchao Zhang<u></u><u></u></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12pt"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:12pt">On Mon, Apr 14, 2025 at 4:39</span><span style="font-size:12pt;font-family:Arial,sans-serif"> </span><span style="font-size:12pt">PM Angus, Justin Ray via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Hello,</span><span style="font-size:12pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span><span style="font-size:12pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">A project I work on uses GMRES via PETSc. In particular, we have had good successes using the Additive Schwarz Method + ILU preconditioner setup using
a CPU-based code. I found online where it is stated that “Parts of most preconditioners run directly on the GPU” (<a href="https://urldefense.us/v3/__https:/petsc.org/release/faq/__;!!G_uCfscf7eWS!bw6qeKcY7MKSvlEgcogdKR7fpjZSOFvka6zfDprUZ_sJHdE-YZmRD6UTqWQW3_uGVBII4P-AG0zaGTLbI67_fQ$" target="_blank">https://petsc.org/release/faq/</a>).
Is ASM + ILU also available for GPU platforms?</span><span style="font-size:12pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> </span><span style="font-size:12pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">-Justin</span><span style="font-size:12pt"><u></u><u></u></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!fQqW7mjnh1VSngks4WuKIejODD259NIRzuXN8kFwXrunsgj2S6ecC6AfmnT8mUT5wQFY3F5kWSLiAoDn6W5U$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>