<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Sure,</p>
<p><br>
</p>
<p>For instance on the below graphic, there is a sequence:</p>
<p><img size="93240" contenttype="image/png" id="img555472" contextid="img942061" tabindex="0" style="max-width: 99.9%; user-select: none;" src="cid:c0738ea8-66fe-4bae-aeb0-3cb66f6bfeb4"><br>
</p>
<p><br>
</p>
<p>Blue: PCApply/spmv_fixup_kernel_v2</p>
<p>Red: D2H copy (8000 bytes)</p>
<p>Grey: MPI_irecv</p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Grey: </span>MPI_isend</p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Grey: </span>MPI_Waitall</p>
<p>Green: H2D copy (8000 bytes)</p>
<p>Blue: PCApply/csmv_v2_partition_kernel</p>
<p><br>
</p>
<p>And this is happening in Hypre code.</p>
<p><br>
</p>
<p>I replaced boomeramg by gamg, and confirm now there is only D2D copy during MPI calls. So the issue is related to Hypre (using 2.33).</p>
<p><br>
</p>
<p>I double checked that PETSc configure enables correctly <span style="font-family: Calibri, Helvetica, sans-serif, serif, EmojiFont; font-size: 16px;">--enable-gpu-aware-mpi into Hypre during build, so I think I should contact Hypre team now.</span></p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, serif, EmojiFont; font-size: 16px;">And switch to gamg for the moment.</span></p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, serif, EmojiFont; font-size: 16px;"><br>
</span></p>
<p><span style="font-family: Calibri, Helvetica, sans-serif, serif, EmojiFont; font-size: 16px;">Thanks for your patience,</span></p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>De :</b> Barry Smith <bsmith@petsc.dev><br>
<b>Envoyé :</b> lundi 1 septembre 2025 00:32:23<br>
<b>À :</b> LEDAC Pierre<br>
<b>Cc :</b> petsc-users@mcs.anl.gov<br>
<b>Objet :</b> Re: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div>
<div><br>
</div>
<div> Can you pinpoint the MPI calls (and the routines in PETSc or hypre that they are in) that are not using CUDA-aware message passing? That is, inside the MPI call, they are copying to host memory and doing the needed inter-process communication from there?
I do not understand the graphics you have sent.</div>
<div><br>
</div>
<div> Barry</div>
<div><br>
</div>
<div>Or is it possible the buffers passed to MPI are not on the GPU and so naturally do the MPI from host memory? If so, where?<br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 31, 2025, at 1:30 PM, LEDAC Pierre <Pierre.LEDAC@cea.fr> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">
<div style="margin-top: 0px; margin-bottom: 0px;">Ok, I just tried <span>--enable-gpu-aware-mpi passed to Hypre, </span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span>Hypre_config.h defines now <span>HYPRE_USING_GPU_AWARE_MPI 1</span></span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;"><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;">But still no D2D copy near MPI calls in ex46.c example. </span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;"><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;">Probably an obvious </span><span style="font-size: 12pt;">thing I forgot during PETSc configure, but I don't see...</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family: Tahoma; font-size: 13px;">
<div class="BodyFragment"><font size="2"><span style="font-size: 10pt;">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 833.96875px;">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span>
<div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br>
<b>Envoyé :</b><span class="Apple-converted-space"> </span>dimanche 31 août 2025 19:13:36<br>
<b>À :</b><span class="Apple-converted-space"> </span>Barry Smith<br>
<b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Objet :</b><span class="Apple-converted-space"> </span>RE: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="margin-top: 0px; margin-bottom: 0px;">Barry,</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">It solved the unrecognized option but still exchanging MPI messages through the host.</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">I switch to a simpler test case without reading a matrix (s<span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">rc/ksp/ksp/tutorials/ex46.c) </span>but
get the same behaviour.<span style="font-size: 12pt;"></span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span>In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc
build. </span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span>I will give a try with <span>--enable-gpu-aware-mpi passed to Hypre.</span></span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Do you know an
example in PETSc which specifically bench with/without C</span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">uda-Aware
enabled for MPI </span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">?</span><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span><span id="cid:93d886d4-28ec-4b73-b602-3a559fffd435"><pastedImage.png></span><br>
</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family: Tahoma; font-size: 13px;">
<div class="BodyFragment"><font size="2"><span style="font-size: 10pt;">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display: inline-block; width: 833.96875px;">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<b>Envoyé :</b><span class="Apple-converted-space"> </span>dimanche 31 août 2025 16:33:38<br>
<b>À :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br>
<b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div>
<div><br>
</div>
Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your
code)?
<div><br>
</div>
<div> Barry</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 31, 2025, at 4:32 AM, LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="margin-top: 0px; margin-bottom: 0px;">Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda)</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span>mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse
-vec_type cuda</span><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span>...</span></div>
<p style="margin-top: 0px; margin-bottom: 0px;"></p>
<div><b>WARNING! There are options you set that were not used!</b></div>
<div><b>WARNING! could be spelling mistake, etc!</b></div>
<div><b>There is one unused database option. It is:</b></div>
<div><b>Option left: name:-vec_type value: cuda source: command lin</b></div>
<p style="margin-top: 0px; margin-bottom: 0px;"></p>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family: Tahoma; font-size: 13px;">
<div class="BodyFragment"><font size="2"><span style="font-size: 10pt;">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; display: inline-block; width: 1014.296875px;">
<span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline !important;"></span>
<div id="divRplyFwdMsg" dir="ltr" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;">
<font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br>
<b>Envoyé :</b><span class="Apple-converted-space"> </span>samedi 30 août 2025 21:47:07<br>
<b>À :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br>
<b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br>
<b>Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] [MPI][GPU]</font>
<div> </div>
</div>
<div style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;">
<div><br>
</div>
Did you try the additional option -vec_type cuda with ex10.c ?
<div><br>
</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Aug 30, 2025, at 1:16 PM, LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="margin-top: 0px; margin-bottom: 0px;">Hello,</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur.</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">I reproduced the PETSc issue with the <span>example under src/ksp/ksp/tutorials/</span>ex10 on 2 MPI ranks. <span style="font-size: 12pt;">See output in ex10.log</span></div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls.</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;">Thanks for your help,</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><span id="cid:fe096b00-e3b7-4aa3-9d78-d8a44adbe145"><pastedImage.png></span><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div style="margin-top: 0px; margin-bottom: 0px;"><br>
</div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<div style="font-family: Tahoma; font-size: 13px;">
<div class="BodyFragment"><font size="2"><span style="font-size: 10pt;">
<div class="PlainText">Pierre LEDAC<br>
Commissariat à l’énergie atomique et aux énergies alternatives<br>
Centre de SACLAY<br>
DES/ISAS/DM2S/SGLS/LCAN<br>
Bâtiment 451 – point courrier n°41<br>
F-91191 Gif-sur-Yvette<br>
+33 1 69 08 04 03<br>
+33 6 83 42 05 79</div>
</span></font></div>
</div>
</div>
</div>
</div>
<span id="cid:D4C34088-D3F6-4F4E-8156-CF22AE4A5C85"><ex10.log></span></div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>