<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 9 May 2019 at 18:19, Matthew Knepley via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, May 9, 2019 at 12:34 PM GIRET Jean-Christophe via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="FR">
<div class="gmail-m_3390898189526249960gmail-m_-8472802224438753808WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks Mark and Jed for your quick answers.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">So the idea is to define all the Vecs on the world communicator, and perform the communications using traditional scatter
objects? The data would still be accessible on the two sub-communicators as they are both subsets of the MPI_COMM_WORLD communicator, but they would be used while creating the Vecs or the IS for the scatter. Is that right?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I’m currently trying, without success, to perform a Scatter from a MPI Vec defined on a subcomm to another Vec defined
on the world comm, and vice-versa. But I don’t know if it’s possible.</span></p></div></div></blockquote><div><br></div><div>You cannot do that. What you want to do is:</div><div><br></div><div>1) Create two Vecs on COMM_WORLD. Make the second vec have all 0 sizes on processes not in the subcomm.</div><div><br></div><div>2) Create a scatter between the two Vec</div><div><br></div><div>3) Scatter data</div><div><br></div><div>4) Use VecGetArray() to get the pointer to the data on the second vec, and use VecCreateWithArray() ONLY on the subcomm,</div><div> or if you do not mind copies, just use a copy.</div></div></div></blockquote><div><br></div><div>You can find a concrete example of doing exactly what Matt described above within PCTELESCOPE.</div><div><br></div><div>See </div><div> src/ksp/pc/impls/telescope/telescope.c</div><div>or go here</div><div> <a href="https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/telescope/telescope.c.html#PCTELESCOPE">https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/pc/impls/telescope/telescope.c.html#PCTELESCOPE</a><br></div><div><br></div><div>Specifically you want to examine the functions</div><div> PCTelescopeSetUp_default()<br></div><div> PCApply_Telescope()<br></div><div><br></div><div>To explain in more detail, within PCTelescopeSetUp_default() </div><div>* I create two vectors, <span style="color:rgb(0,0,0)">xtmp (living on comm_1 with some ranks owning zero entries) and xred (defined on a sub comm of comm_1, say sub_comm_1).</span></div><div><span style="color:rgb(0,0,0)">* I create the scatter between some input x and xtmp.</span></div><div><span style="color:rgb(0,0,0)">The points implement Matt's steps 1 & 2</span></div><div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">In </span>PCApply_Telescope(), I perform the scatter between the input arg x and the vector xtmp (both defined on comm_1) [Matt's step 3]. </div><div><br></div><div>Then you'll see this</div><div><br></div><div><div>if (xred) {</div><div> PetscScalar *LA_xred;</div><div> VecGetOwnershipRange(xred,&st,&ed);</div><div> VecGetArray(xred,&LA_xred);</div><div> for (i=0; i<ed-st; i++) {</div><div> LA_xred[i] = x_array[i];</div><div> }</div><div> VecRestoreArray(xred,&LA_xred);</div><div> }</div></div><div><br></div><div>This is Matt's step 4. Note, that xred was created such that it was NULL on ranks in comm_1 which are not defined in <span style="color:rgb(0,0,0)">sub_comm_1. Hence the test </span>if (xred) is checking if we are living on a rank within sub_comm_1. If this is true, I copy ALL entries from xtmp. </div><div><br></div><div>In PCTELESCOPE I did not bother with the optimization of using VecCreateWithArray() - I just do the copy.</div><div><br></div><div><br></div><div>Cheers</div><div>Dave</div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="FR"><div class="gmail-m_3390898189526249960gmail-m_-8472802224438753808WordSection1"><p class="MsoNormal"> </p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="FR"><div class="gmail-m_3390898189526249960gmail-m_-8472802224438753808WordSection1"><p class="MsoNormal"></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I can imagine that trying doing that seems a bit strange. However, I’m dealing with code coupling (and linear algebra
for the main part of the code), and my idea was trying to use the Vec data structures to perform data exchange between some parts of the software which would have their own communicator. It would eliminate the need to re-implement an ad-hoc solution.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">An option would be to stick on the world communicator for all the PETSc part, but I could face some situations where my
Vecs could be small while I would have to run the whole simulation on an important number of core for the coupled part. I imagine that It may not really serve the linear system solving part in terms of performance. Another one would be perform all the PETSc
operations on a sub-communicator and use “raw” MPI communications between the communicators to perform the data exchange for the coupling part.
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks again for your support,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Jean-Christophe<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">De :</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Mark Adams [mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>]
<br>
<b>Envoyé :</b> mardi 7 mai 2019 21:39<br>
<b>À :</b> GIRET Jean-Christophe<br>
<b>Cc :</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Objet :</b> Re: [petsc-users] Question about parallel Vectors and communicators<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, May 7, 2019 at 11:38 AM GIRET Jean-Christophe via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Dear PETSc users,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I would like to use Petsc4Py for a project extension, which consists mainly of:</span><u></u><u></u></p>
<p class="gmail-m_3390898189526249960gmail-m_-8472802224438753808gmail-m5400229876258405851msolistparagraph" style="margin-left:27pt">
<span lang="EN-US">-</span><span lang="EN-US" style="font-size:7pt"> </span>
<span lang="EN-US">Storing data and matrices on several rank/nodes which could not fit on a single node.</span><u></u><u></u></p>
<p class="gmail-m_3390898189526249960gmail-m_-8472802224438753808gmail-m5400229876258405851msolistparagraph" style="margin-left:27pt">
<span lang="EN-US">-</span><span lang="EN-US" style="font-size:7pt"> </span>
<span lang="EN-US">Performing some linear algebra in a parallel fashion (solving sparse linear system for instance)</span><u></u><u></u></p>
<p class="gmail-m_3390898189526249960gmail-m_-8472802224438753808gmail-m5400229876258405851msolistparagraph" style="margin-left:27pt">
<span lang="EN-US">-</span><span lang="EN-US" style="font-size:7pt"> </span>
<span lang="EN-US">Exchanging those data structures (parallel vectors) between non-overlapping MPI communicators, created for instance by splitting MPI_COMM_WORLD.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">While the two first items seems to be well addressed by PETSc, I am wondering about the last one.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Is it possible to access the data of a vector, defined on a communicator from another, non-overlapping communicator? From what I have seen from the documentation
and the several threads on the user mailing-list, I would say no. But maybe I am missing something? If not, is it possible to transfer a vector defined on a given communicator on a communicator which is a subset of the previous one?</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you are sending to a subset of processes then VecGetSubVec + Jed's tricks might work.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Best regards,
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Jean-Christophe</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_3390898189526249960gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div></div></div></div></div></div></div></div></div></div></div>