<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Thanks Barry, I agree but didn't dare asking for that.</p>
<br>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> Barry Smith <bsmith@petsc.dev><br>
<b>Envoyé :</b> mercredi 30 juillet 2025 20:34:26<br>
<b>À :</b> Junchao Zhang<br>
<b>Cc :</b> LEDAC Pierre; petsc-users@mcs.anl.gov<br>
<b>Objet :</b> Re: [petsc-users] [GPU] Jacobi preconditioner</font>
<div> </div>
</div>
<div>
<div><br>
</div>
We absolutely should have a <font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols" size="3">MatGetDiagonal_SeqAIJCUSPARSE(). It's somewhat embarrassing that
we don't provide this.</font>
<div><br>
</div>
<div> I have found some potential code at <a href="https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!flO1UCfj-bia4eeLdSw3qZ5b15r6I7UIktvoIFPaqwGfdbGlABa_9JjiwW6xy6Gan0s-kA6hRXDz3jjsoCZnkf_SlbiT$">https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse</a></div>
<div><br>
</div>
<div> Barry</div>
<div><br>
</div>
<div><br>
</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Jul 28, 2025, at 11:43 AM, Junchao Zhang <junchao.zhang@gmail.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="ltr">Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda and petsc/kokkos backends are separate code. <br>
If petsc/kokkos meet your needs, then just use them. For petsc users, we hope it will be just a difference of extra --download-kokkos --download-kokkos-kernels in configuration.
<div><br>
</div>
<div>--Junchao Zhang</div>
<br>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Mon, Jul 28, 2025 at 2:51 AM LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="msg3805772366042246901">
<div dir="ltr">
<div id="m_-58953184740190291divtagdefaultwrapper" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;" dir="ltr">
<p>Hello all,</p>
<p><br>
</p>
<p>We are solving with PETSc a linear system updated every time step (constant stencil but coefficients changing).</p>
<p><br>
</p>
<p>The matrix is preallocated once with <span>MatSetPreallocationCOO() then filled each time step with <span>MatSetValuesCOO() and we use device pointers for coo_i, coo_j, and coefficients values.</span></span></p>
<p><span><span><br>
</span></span></p>
<p><span><span>It is working fine with a GMRES Ksp solver and PC Jacobi but we are surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. Looking at the API, it seems there is no <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">MatGetDiagonal_SeqAIJCUSPARSE()
but a <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">MatGetDiagonal_SeqAIJKOKKOS().</span></span></span></span></p>
<p><br>
</p>
<p>Does it mean we should use Kokkos backend in PETSc to have Jacobi preconditioner built directly on device ? Or I am doing something wrong ?</p>
<p>NB: Gmres is running well on device.</p>
<p><br>
</p>
<p>I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each solve on host but it increases significantly the number of iterations.</p>
<p><br>
</p>
<p>Thanks,</p>
<p><br>
</p>
<p><span id="cid:ii_19851ad63faf456b1e51"><pastedImage.png></span><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div id="m_-58953184740190291Signature">
<div id="m_-58953184740190291divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family:Tahoma;font-size:13px">
<div><font size="2"><span style="font-size:10pt">
<div>Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>