<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Then Jacob has to come back and finish the baking.<div><br></div><div> Barry</div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 4, 2025, at 11:27 PM, Junchao Zhang <junchao.zhang@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div>CUPM is half baked, and only supports Vec and MATDENSE. </div><div><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Aug 4, 2025 at 7:31 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> I thought the CUPM code was intended to allow us to have common code between NVIDIA and AMD?<div><br id="m_1734974551534555438lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jul 31, 2025, at 11:05 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:</div><br><div><div dir="ltr"><div dir="ltr"><div>What would embarrass me more is to copy the same code to <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt">MatGetDiagonal_</span><span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:12pt">SeqAIJHIPSPARSE.</span></div><div><br></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 30, 2025 at 1:34 PM Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> We absolutely should have a <font face="Calibri, Helvetica, sans-serif, EmojiFont, Apple Color Emoji, Segoe UI Emoji, NotoColorEmoji, Segoe UI Symbol, Android Emoji, EmojiSymbols" size="3">MatGetDiagonal_SeqAIJCUSPARSE(). It's somewhat embarrassing that we don't provide this.</font><div><br></div><div> I have found some potential code at <a href="https://urldefense.us/v3/__https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse__;!!G_uCfscf7eWS!dkLsir2oikTcYGJKRYgXdA9kTWZzg2MXrL0j0w750Ji-8wfT-NJG6B0cTCgmxasEkr_G8Fb3gHEi7xsu4I1_bao$" target="_blank">https://stackoverflow.com/questions/60311408/how-to-get-the-diagonal-of-a-sparse-matrix-in-cusparse</a></div><div><br></div><div> Barry</div><div><br></div><div><br></div><div><br id="m_1734974551534555438m_2893945046185193653lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jul 28, 2025, at 11:43 AM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>> wrote:</div><br><div><div dir="ltr">Yes, MatGetDiagonal_SeqAIJCUSPARSE hasn't been implemented. petsc/cuda and petsc/kokkos backends are separate code. <br>If petsc/kokkos meet your needs, then just use them. For petsc users, we hope it will be just a difference of extra --download-kokkos --download-kokkos-kernels in configuration. <div><br></div><div>--Junchao Zhang</div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 28, 2025 at 2:51 AM LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr" target="_blank">Pierre.LEDAC@cea.fr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div id="m_1734974551534555438m_2893945046185193653m_-58953184740190291divtagdefaultwrapper" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif" dir="ltr"><p>Hello all,</p><p><br>
</p><p>We are solving with PETSc a linear system updated every time step (constant stencil but coefficients changing).</p><p><br>
</p><p>The matrix is preallocated once with <span>MatSetPreallocationCOO() then filled each time step with <span>MatSetValuesCOO() and we use device pointers for coo_i, coo_j, and coefficients values.</span></span></p><p><span><span><br>
</span></span></p><p><span><span>It is working fine with a GMRES Ksp solver and PC Jacobi but we are surprised to see that every time step, during PCSetUp, MatGetDiagonal_SeqAIJ is called whereas the matrix is on the device. Looking at the API, it seems there is no <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">MatGetDiagonal_SeqAIJCUSPARSE()
but a <span style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols;font-size:16px">MatGetDiagonal_SeqAIJKOKKOS().</span></span></span></span></p><p><br>
</p><p>Does it mean we should use Kokkos backend in PETSc to have Jacobi preconditioner built directly on device ? Or I am doing something wrong ?</p><p>NB: Gmres is running well on device.</p><p><br>
</p><p>I could use -ksp_reuse_preconditioner to avoid Jacobi being recreated each solve on host but it increases significantly the number of iterations.</p><p><br>
</p><p>Thanks,</p><p><br>
</p><p><span id="m_1734974551534555438m_2893945046185193653cid:ii_19851ad63faf456b1e51"><pastedImage.png></span><br>
</p><p><br>
</p><p><br>
</p>
<div id="m_1734974551534555438m_2893945046185193653m_-58953184740190291Signature">
<div id="m_1734974551534555438m_2893945046185193653m_-58953184740190291divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="font-family:Tahoma;font-size:13px">
<div><font size="2"><span style="font-size:10pt">
<div>Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div></div>
</div></blockquote></div><br></div></div></blockquote></div>
</div></blockquote></div><br></div></body></html>