<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div><div> Can you pinpoint the MPI calls (and the routines in PETSc or hypre that they are in) that are not using CUDA-aware message passing? That is, inside the MPI call, they are copying to host memory and doing the needed inter-process communication from there? I do not understand the graphics you have sent.</div><div><br></div><div> Barry</div><div><br></div><div>Or is it possible the buffers passed to MPI are not on the GPU and so naturally do the MPI from host memory? If so, where?<br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 31, 2025, at 1:30 PM, LEDAC Pierre <Pierre.LEDAC@cea.fr> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">Ok, I just tried <span>--enable-gpu-aware-mpi passed to Hypre, </span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>Hypre_config.h defines now <span>HYPRE_USING_GPU_AWARE_MPI 1</span></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;"><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;">But still no D2D copy near MPI calls in ex46.c example. </span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;"><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-size: 12pt;">Probably an obvious </span><span style="font-size: 12pt;">thing I forgot during PETSc configure, but I don't see...</span></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div id="Signature"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="font-family: Tahoma; font-size: 13px;"><div class="BodyFragment"><font size="2"><span style="font-size: 10pt;"><div class="PlainText">Pierre LEDAC<br>Commissariat à l’énergie atomique et aux énergies alternatives<br>Centre de SACLAY<br>DES/ISAS/DM2S/SGLS/LCAN<br>Bâtiment 451 – point courrier n°41<br>F-91191 Gif-sur-Yvette<br>+33 1 69 08 04 03<br>+33 6 83 42 05 79</div></span></font></div></div></div></div></div><hr tabindex="-1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 833.96875px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br><b>Envoyé :</b><span class="Apple-converted-space"> </span>dimanche 31 août 2025 19:13:36<br><b>À :</b><span class="Apple-converted-space"> </span>Barry Smith<br><b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br><b>Objet :</b><span class="Apple-converted-space"> </span>RE: [petsc-users] [MPI][GPU]</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="margin-top: 0px; margin-bottom: 0px;">Barry,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">It solved the unrecognized option but still exchanging MPI messages through the host.</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">I switch to a simpler test case without reading a matrix (s<span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">rc/ksp/ksp/tutorials/ex46.c) </span>but get the same behaviour.<span style="font-size: 12pt;"></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>In the Nsys profile for ex46, the MPI synchronizations occurs during PCApply so now I am wondering if the issue is related to the fact than Hypre is not configured/enabled with MPI GPU-Aware in the PETSc build. </span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>I will give a try with <span>--enable-gpu-aware-mpi passed to Hypre.</span></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">Do you know an example in PETSc which specifically bench with/without C</span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">uda-Aware enabled for MPI </span><span style="font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">?</span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><span><span id="cid:93d886d4-28ec-4b73-b602-3a559fffd435"><pastedImage.png></span><br></span></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div id="Signature"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="font-family: Tahoma; font-size: 13px;"><div class="BodyFragment"><font size="2"><span style="font-size: 10pt;"><div class="PlainText">Pierre LEDAC<br>Commissariat à l’énergie atomique et aux énergies alternatives<br>Centre de SACLAY<br>DES/ISAS/DM2S/SGLS/LCAN<br>Bâtiment 451 – point courrier n°41<br>F-91191 Gif-sur-Yvette<br>+33 1 69 08 04 03<br>+33 6 83 42 05 79</div></span></font></div></div></div></div></div><hr tabindex="-1" style="display: inline-block; width: 833.96875px;"><div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Envoyé :</b><span class="Apple-converted-space"> </span>dimanche 31 août 2025 16:33:38<br><b>À :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br><b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br><b>Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] [MPI][GPU]</font><div> </div></div><div><div><br></div> Ahh, that ex10.c is missing a VecSetFromOptions() call before the VecLoad() and friends. In contrast, the matrix has a MatSetFromOptions(). Can you try adding it to ex10.c and see if that resolves the problem with ex10.c (and may be a path forward for your code)?<div><br></div><div> Barry</div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 31, 2025, at 4:32 AM, LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="margin-top: 0px; margin-bottom: 0px;">Yes, but was surprised it was not used, so I removed it (same for -vec_type mpicuda)</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>mpirun -np 2 ./ex10 2 -f Matrix_3133717_rows_1_cpus.petsc -ksp_view -log_view -ksp_monitor -ksp_type cg -pc_type hypre -pc_hypre_type boomeramg -pc_hypre_boomeramg_strong_threshold 0.7 -mat_type aijcusparse -vec_type cuda</span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><span>...</span></div><p style="margin-top: 0px; margin-bottom: 0px;"></p><div><b>WARNING! There are options you set that were not used!</b></div><div><b>WARNING! could be spelling mistake, etc!</b></div><div><b>There is one unused database option. It is:</b></div><div><b>Option left: name:-vec_type value: cuda source: command lin</b></div><p style="margin-top: 0px; margin-bottom: 0px;"></p><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div id="Signature"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="font-family: Tahoma; font-size: 13px;"><div class="BodyFragment"><font size="2"><span style="font-size: 10pt;"><div class="PlainText">Pierre LEDAC<br>Commissariat à l’énergie atomique et aux énergies alternatives<br>Centre de SACLAY<br>DES/ISAS/DM2S/SGLS/LCAN<br>Bâtiment 451 – point courrier n°41<br>F-91191 Gif-sur-Yvette<br>+33 1 69 08 04 03<br>+33 6 83 42 05 79</div></span></font></div></div></div></div></div><hr tabindex="-1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; display: inline-block; width: 1014.296875px;"><span style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>De :</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Envoyé :</b><span class="Apple-converted-space"> </span>samedi 30 août 2025 21:47:07<br><b>À :</b><span class="Apple-converted-space"> </span>LEDAC Pierre<br><b>Cc :</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><br><b>Objet :</b><span class="Apple-converted-space"> </span>Re: [petsc-users] [MPI][GPU]</font><div> </div></div><div style="font-family: Helvetica; font-size: 18px; font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div><br></div>Did you try the additional option -vec_type cuda with ex10.c ?<div><br></div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 30, 2025, at 1:16 PM, LEDAC Pierre <<a href="mailto:Pierre.LEDAC@cea.fr">Pierre.LEDAC@cea.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="margin-top: 0px; margin-bottom: 0px;">Hello,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur.</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">I reproduced the PETSc issue with the <span>example under src/ksp/ksp/tutorials/</span>ex10 on 2 MPI ranks. <span style="font-size: 12pt;">See output in ex10.log</span></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls.</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Thanks for your help,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><span id="cid:fe096b00-e3b7-4aa3-9d78-d8a44adbe145"><pastedImage.png></span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div id="Signature"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="font-family: Tahoma; font-size: 13px;"><div class="BodyFragment"><font size="2"><span style="font-size: 10pt;"><div class="PlainText">Pierre LEDAC<br>Commissariat à l’énergie atomique et aux énergies alternatives<br>Centre de SACLAY<br>DES/ISAS/DM2S/SGLS/LCAN<br>Bâtiment 451 – point courrier n°41<br>F-91191 Gif-sur-Yvette<br>+33 1 69 08 04 03<br>+33 6 83 42 05 79</div></span></font></div></div></div></div></div><span id="cid:D4C34088-D3F6-4F4E-8156-CF22AE4A5C85"><ex10.log></span></div></blockquote></div></div></div></div></blockquote></div></div></div></div></div></div></blockquote></div><br></div></body></html>