<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div>Did you try the additional option -vec_type cuda with ex10.c ?<div><br></div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 30, 2025, at 1:16 PM, LEDAC Pierre <Pierre.LEDAC@cea.fr> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div id="divtagdefaultwrapper" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;"><div style="margin-top: 0px; margin-bottom: 0px;">Hello,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">My code is built with PETSc 3.23+OpenMPI 4.1.6 (Cuda support enabled) and profling indicates that MPI communications are done between GPUs in all the code except PETSc part where D2H transfers occur.</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">I reproduced the PETSc issue with the <span>example under src/ksp/ksp/tutorials/</span>ex10 on 2 MPI ranks. <span style="font-size: 12pt;">See output in ex10.log</span></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Also below the Nsys system profiling on ex10 with D2H and H2D copies before/after MPI calls.</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;">Thanks for your help,</div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><span id="cid:fe096b00-e3b7-4aa3-9d78-d8a44adbe145"><pastedImage.png></span><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div style="margin-top: 0px; margin-bottom: 0px;"><br></div><div id="Signature"><div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;"><div style="font-family: Tahoma; font-size: 13px;"><div class="BodyFragment"><font size="2"><span style="font-size: 10pt;"><div class="PlainText">Pierre LEDAC<br>Commissariat à l’énergie atomique et aux énergies alternatives<br>Centre de SACLAY<br>DES/ISAS/DM2S/SGLS/LCAN<br>Bâtiment 451 – point courrier n°41<br>F-91191 Gif-sur-Yvette<br>+33 1 69 08 04 03<br>+33 6 83 42 05 79</div></span></font></div></div></div></div></div><span id="cid:D4C34088-D3F6-4F4E-8156-CF22AE4A5C85"><ex10.log></span></div></blockquote></div><br></div></body></html>