<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Ok, I just tried <span>--enable-gpu-aware-mpi passed to Hypre, </span></p>
<p><span><br>
</span></p>
<p><span>Hypre_config.h defines now <span>HYPRE_USING_GPU_AWARE_MPI 1</span></span></p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">But still no D2D copy near MPI calls in ex46.c example. </span></p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">Probably an obvious </span><span style="font-size: 12pt;">thing I forgot during PETSc configure, but I don't see...</span></p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> LEDAC Pierre<br>
<b>Envoyé :</b> dimanche 31 août 2025 19:13:36<br>
<b>À :</b> Barry Smith<br>
<b>Cc :</b> petsc-users@mcs.anl.gov<br>
<b>Objet :</b> RE: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p>Barry,</p>
<p><br>
</p>
<p>It solved the unrecognized option but still exchanging MPI messages through the host.</p>
<p><br>
</p>
<p>I switch to a simpler test case without reading a matrix (s<span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">rc/ksp/ksp/tutorials/ex46.c) </span>but
get the same behaviour.<span style="font-size: 12pt;"></span></p>
<p><span><br>
</span></p>
<p><span>In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc build. </span></p>
<p><span>I will give a try with <span>--enable-gpu-aware-mpi passed to Hypre.</span></span></p>
<p><span><br>
</span></p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Do you know an example in PETSc which specifically bench with/without
C</span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">uda-Aware enabled for MPI </span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">?</span><br>
</p>
<p><span><br>
</span></p>
<p><span><img size="65351" contenttype="image/png" id="img636399" contextid="img771391" tabindex="0" style="max-width: 99.9%; user-select: none;" src="cid:93d886d4-28ec-4b73-b602-3a559fffd435"><br>
</span></p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>De :</b> Barry Smith <bsmith@petsc.dev><br>
<b>Envoyé :</b> dimanche 31 août 2025 16:33:38<br>
<b>À :</b> LEDAC Pierre<br>
<b>Cc :</b> petsc-users@mcs.anl.gov<br>
<b>Objet :</b> Re: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div>
<div><br>
</div>
Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your
code)?
<div><br>
</div>
<div> Barry</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 31, 2025, at 4:32 AM, LEDAC Pierre <Pierre.LEDAC@cea.fr> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="margin-top:0px; margin-bottom:0px">Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda)</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><span>mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse
-vec_type cuda</span><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><span>...</span></div>
<p style="margin-top:0px; margin-bottom:0px"></p>
<div><b>WARNING! There are options you set that were not used!</b></div>
<div><b>WARNING! could be spelling mistake, etc!</b></div>
<div><b>There is one unused database option. It is:</b></div>
<div><b>Option left: name:-vec_type value: cuda source: command lin</b></div>
<p style="margin-top:0px; margin-bottom:0px"></p>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; display:inline-block; width:1014.296875px">
<span style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="divRplyFwdMsg" dir="ltr" style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>De :</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<b>Envoyé :</b><span class="Apple-converted-space"> </span>samedi 30 août 2025 21:47:07<br>
<b>À :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br>
<b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div style="font-family:Helvetica; font-size:18px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div><br>
</div>
Did you try the additional option -vec_type cuda with ex10.c ?
<div><br>
</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 30, 2025, at 1:16 PM, LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="margin-top:0px; margin-bottom:0px">Hello,</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px">My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur.</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px">I reproduced the PETSc issue with the <span>example under src/ksp/ksp/tutorials/</span>ex10 on 2 MPI ranks. <span style="font-size:12pt">See output in ex10.log</span></div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px">Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls.</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px">Thanks for your help,</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><span id="cid:fe096b00-e3b7-4aa3-9d78-d8a44adbe145"><pastedImage.png></span><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<span id="cid:D4C34088-D3F6-4F4E-8156-CF22AE4A5C85"><ex10.log></span></div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>