<div dir="ltr"><div dir="ltr">On Wed, Sep 21, 2022 at 10:35 AM feng wang <<a href="mailto:snailsoar@hotmail.com">snailsoar@hotmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg6847406593868348465">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
For your 2<span><sup>nd</sup> suggestion on halo exchange, I get the idea and roughly know how to do it, but there are some implementation details which I am not quite sure.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>If I understand it correctly, in <font size="2"><span style="font-size:11pt">MatMult</span></font>(Mat m ,Vec x, Vec y), Vec
<b>x</b> is a normal parallel vector and it does not contain halo values. Suppose I create an
<font size="2"><span style="font-size:11pt">auxiliary</span></font> ghost vector <b>
x_g</b>, then I assign the values of <b>x</b> to <b>x_g</b>. The values of the halo for each partition will not be assigned at this stage.
<br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>But If I call VecGhostUpdateBegin/End(<b>x_g</b>, INSERT_VALUES, SCATTER_FORWARD), this will fill the values of the halo cells of
<b>x_g </b>for each partition. Then <b>x_g</b> has local and halo cells assigned correctly and I can use
<b>x_g</b> to do my computation. Is this what you mean?</span></div></div></div></blockquote><div><br></div><div>Yes</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg6847406593868348465"><div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>Thanks,</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span>Feng<br>
</span></div>
<div id="m_9167181084313841329signature_bookmark"></div>
<div id="m_9167181084313841329appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_9167181084313841329divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>><br>
<b>Sent:</b> 21 September 2022 13:07<br>
<b>To:</b> feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div><br>
<br>
> El 21 sept 2022, a las 14:47, feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> escribió:<br>
> <br>
> Thanks Jose, I will try this and will come back to this thread if I have any issue.<br>
> <br>
> Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards?<br>
<br>
Eigenvectors are stored in parallel vectors, which are used in subsequent parallel computation in most applications. If for some reason you need to gather them in a single MPI process you can use e.g. VecScatterCreateToZero()<br>
<br>
> <br>
> Thanks,<br>
> Feng<br>
> <br>
> From: Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank">jroman@dsic.upv.es</a>><br>
> Sent: 21 September 2022 12:34<br>
> To: feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
> Cc: Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
> Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange<br>
> <br>
> If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work.<br>
> <br>
> A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result
in the regular parallel vector y.<br>
> <br>
> Jose<br>
> <br>
> <br>
> > El 21 sept 2022, a las 14:09, feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> escribió:<br>
> > <br>
> > Thanks for your reply. <br>
> > <br>
> > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how
many ghost cells there are?<br>
> > <br>
> > Thanks,<br>
> > Feng<br>
> > From: Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
> > Sent: 21 September 2022 11:58<br>
> > To: feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>><br>
> > Cc: <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
> > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange<br>
> > <br>
> > On Wed, Sep 21, 2022 at 7:41 AM feng wang <<a href="mailto:snailsoar@hotmail.com" target="_blank">snailsoar@hotmail.com</a>> wrote:<br>
> > Hello,<br>
> > <br>
> > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel.<br>
> > <br>
> > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then
used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc?<br>
> > <br>
> > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult().<br>
> > <br>
> > Thanks,<br>
> > <br>
> > Matt<br>
> > <br>
> > Thanks,<br>
> > Feng <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > -- <br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> > <br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</div>
</span></font></div>
</div>
</div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>