<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Mat,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I've tried the suggested approach. The halo cells are not exchanged somehow. Below is how I do it, have I missed anything?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted5 ContentPasted6">
I create a ghost vector <i>petsc_dcsv</i> and it is a data member of the class cFdDomain, which is a context of the shell matrix.
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted4">
<b><i> PetscCall(VecCreateGhostBlock(*A_COMM_WORLD, blocksize, blocksize*nlocal, PETSC_DECIDE ,nghost, ighost, &petsc_dcsv));</i></b><br class="ContentPasted4">
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted4 ContentPasted7">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted4 ContentPasted7">
blocksize and nv have the same value. nlocal is number of local cells and nghost is number of halo cells. ighost contains the ghost cell index.<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted4">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Below is how I compute a matrix-vector product with a shell matrix<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<b><br>
</b></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<b><i> PetscErrorCode cFdDomain::mymult_slepc(Mat m ,Vec x, Vec y)</i></b>
<div class="ContentPasted0"><b><i> {</i></b></div>
<div class="ContentPasted0"><b><i> void *ctx;</i></b></div>
<div class="ContentPasted0"><b><i> cFdDomain *myctx;</i></b></div>
<div class="ContentPasted0"><b><i> PetscErrorCode ierr;</i></b></div>
<div><b><i><br class="ContentPasted0">
</i></b></div>
<div class="ContentPasted0"><b><i> MatShellGetContext(m, &ctx);</i></b></div>
<div class="ContentPasted0"><b><i> myctx = (cFdDomain*)ctx;</i></b></div>
<div><b><i><br class="ContentPasted0">
</i></b></div>
<div class="ContentPasted0"><b><i>//matrix-vector product</i></b></div>
<div class="ContentPasted0"><b><i> ierr = myctx->myfunc(x, y); CHKERRQ(ierr);</i></b></div>
<div><b><i><br class="ContentPasted0">
</i></b></div>
<b><i> ierr = 0;</i></b>
<div class="ContentPasted0"><b><i> return ierr;</i></b></div>
<div class="ContentPasted0"><b><i> }</i></b></div>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<b><i> PetscErrorCode cFdDomain::myfunc(Vec in, Vec out)</i></b>
<div class="ContentPasted1"><b><i> {</i></b></div>
<b><i>//some declaration<br>
</i></b>
<div><b><i><br class="ContentPasted1">
</i></b></div>
<div class="ContentPasted1"><b><i> ierr = VecGetArray(petsc_dcsv,&array_g); CHKERRQ(ierr);</i></b></div>
<div class="ContentPasted1"><b><i> ierr = VecGetArrayRead(in, &array); CHKERRQ(ierr);</i></b></div>
<div><b><i><br class="ContentPasted1">
</i></b></div>
<div class="ContentPasted1"><b><i> //assign in to petsc_dcsv, only local cells</i></b></div>
<div class="ContentPasted1"><b><i> for(iv=0; iv<nv; iv++)</i></b></div>
<div class="ContentPasted1"><b><i> {</i></b></div>
<div class="ContentPasted1"><b><i> for(iq=0; iq<nlocal; iq++)</i></b></div>
<div class="ContentPasted1"><b><i> {</i></b></div>
<div class="ContentPasted1"><b><i> array_g[iv+nv*iq] = array[iv + nv*iq];</i></b></div>
<div class="ContentPasted1"><b><i> }</i></b></div>
<div class="ContentPasted1"><b><i> }</i></b></div>
<div><b><i><br class="ContentPasted1">
</i></b></div>
<div class="ContentPasted1"><b><i> ierr = VecRestoreArray(petsc_dcsv,&array_g); CHKERRQ(ierr);</i></b></div>
<div class="ContentPasted1"><b><i> ierr = VecRestoreArrayRead(in, &array); CHKERRQ(ierr);</i></b></div>
<div><b><i><br class="ContentPasted1">
</i></b></div>
<div class="ContentPasted1"><b><i> //update halo cells?</i></b></div>
<div class="ContentPasted1"><b><i> PetscCall(VecGhostUpdateBegin(petsc_dcsv, INSERT_VALUES, SCATTER_FORWARD));</i></b></div>
<div class="ContentPasted1"><b><i> PetscCall(VecGhostUpdateEnd(petsc_dcsv, INSERT_VALUES, SCATTER_FORWARD));</i></b></div>
<div class="ContentPasted1"><b><i> PetscCall(VecGhostGetLocalForm(petsc_dcsv,&veclocal));</i></b></div>
<b><i><br>
</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted2">
<b><i>//read in v</i></b>
<div class="ContentPasted2"><b><i> ierr = VecGetArray(veclocal,&array_ghost); CHKERRQ(ierr);</i></b></div>
<div class="ContentPasted2"><b><i> for(iv=0; iv<nv; iv++)</i></b></div>
<div class="ContentPasted2"><b><i> {</i></b></div>
<div class="ContentPasted2"><b><i> for(iq=0; iq<nlocal; iq++)</i></b></div>
<div class="ContentPasted2"><b><i> {</i></b></div>
<div class="ContentPasted2"><b><i> jq = ilocal[iq];</i></b></div>
<div class="ContentPasted2"><b><i> dq[iv][jq] = array_ghost[iv + nv*iq];</i></b></div>
<div class="ContentPasted2"><b><i> }</i></b></div>
<div><b><i><br class="ContentPasted2">
</i></b></div>
<div class="ContentPasted2"><b><i> for(iq=nlocal; iq<nlocal+nghost; iq++)</i></b></div>
<div class="ContentPasted2"><b><i> {</i></b></div>
<div class="ContentPasted2"><b><i> jq = ighost_local[iq-nlocal];</i></b></div>
<div class="ContentPasted2"><b><i> dq[iv][jq] = array_ghost[iv + nv*iq];</i></b></div>
<div class="ContentPasted2"><b><i> }</i></b></div>
<div class="ContentPasted2"><b><i> }</i></b></div>
<div class="ContentPasted2"><b><i> ierr = VecRestoreArray(veclocal,&array_ghost); CHKERRQ(ierr);</i></b></div>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted2">
<b><i><br>
</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted2">
<b><i> //some computations<br>
</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted2">
<b><i><br>
</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<b><i> PetscCall(VecGhostRestoreLocalForm(petsc_dcsv,&veclocal));<br class="ContentPasted3">
</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<b><i> }</i></b></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<i><br>
</i></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<i><br>
</i></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
so I fill the local part of the ghost vector <i class="ContentPasted8">petsc_dcsv</i> for each rank and then call ghost update, and think this will update the halo cells. it seems not doing that.
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
Thanks,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
Feng<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)" class="elementToProof ContentPasted3">
<i><br>
</i></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b> 21 September 2022 14:36<br>
<b>To:</b> feng wang <snailsoar@hotmail.com><br>
<b>Cc:</b> Jose E. Roman <jroman@dsic.upv.es>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Wed, Sep 21, 2022 at 10:35 AM feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>> wrote:<br>
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg6847406593868348465">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Jose,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
For your 2<span><sup>nd</sup> suggestion on halo exchange, I get the idea and roughly know how to do it, but there are some implementation details which I am not quite sure.</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>If I understand it correctly, in <font size="2"><span style="font-size:11pt">MatMult</span></font>(Mat m ,Vec x, Vec y), Vec
<b>x</b> is a normal parallel vector and it does not contain halo values. Suppose I create an
<font size="2"><span style="font-size:11pt">auxiliary</span></font> ghost vector <b>
x_g</b>, then I assign the values of <b>x</b> to <b>x_g</b>. The values of the halo for each partition will not be assigned at this stage.
<br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>But If I call VecGhostUpdateBegin/End(<b>x_g</b>, INSERT_VALUES, SCATTER_FORWARD), this will fill the values of the halo cells of
<b>x_g </b>for each partition. Then <b>x_g</b> has local and halo cells assigned correctly and I can use
<b>x_g</b> to do my computation. Is this what you mean?</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yes</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg6847406593868348465">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>Thanks,</span></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>Feng<br>
</span></div>
<div id="x_m_9167181084313841329signature_bookmark"></div>
<div id="x_m_9167181084313841329appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_m_9167181084313841329divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" data-auth="NotApplicable">jroman@dsic.upv.es</a>><br>
<b>Sent:</b> 21 September 2022 13:07<br>
<b>To:</b> feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>><br>
<b>Cc:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" data-auth="NotApplicable">knepley@gmail.com</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div><br>
<br>
> El 21 sept 2022, a las 14:47, feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>> escribió:<br>
> <br>
> Thanks Jose, I will try this and will come back to this thread if I have any issue.<br>
> <br>
> Besides, for EPSGetEigenpair, I guess each rank gets its portion of the eigenvector, and I need to put them together afterwards?<br>
<br>
Eigenvectors are stored in parallel vectors, which are used in subsequent parallel computation in most applications. If for some reason you need to gather them in a single MPI process you can use e.g. VecScatterCreateToZero()<br>
<br>
> <br>
> Thanks,<br>
> Feng<br>
> <br>
> From: Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" data-auth="NotApplicable">jroman@dsic.upv.es</a>><br>
> Sent: 21 September 2022 12:34<br>
> To: feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>><br>
> Cc: Matthew Knepley <<a href="mailto:knepley@gmail.com" data-auth="NotApplicable">knepley@gmail.com</a>>;
<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>><br>
> Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange<br>
> <br>
> If you define the MATOP_CREATE_VECS operation in your shell matrix so that it creates a ghost vector, then all vectors within EPS will be ghost vectors, including those that are received as arguments of MatMult(). Not sure if this will work.<br>
> <br>
> A simpler solution is that you store a ghost vector in the context of your shell matrix, and then in MatMult() you receive a regular parallel vector x, then update the ghost points using the auxiliary ghost vector, do the computation and store the result
in the regular parallel vector y.<br>
> <br>
> Jose<br>
> <br>
> <br>
> > El 21 sept 2022, a las 14:09, feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>> escribió:<br>
> > <br>
> > Thanks for your reply. <br>
> > <br>
> > For GMRES, I create a ghost vector and give it to KSPSolve. For Slepc, it only takes the shell matrix for EPSSetOperators. Suppose the shell matrix of the eigensolver defines MatMult(Mat m ,Vec x, Vec y), how does it know Vec x is a ghost vector and how
many ghost cells there are?<br>
> > <br>
> > Thanks,<br>
> > Feng<br>
> > From: Matthew Knepley <<a href="mailto:knepley@gmail.com" data-auth="NotApplicable">knepley@gmail.com</a>><br>
> > Sent: 21 September 2022 11:58<br>
> > To: feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>><br>
> > Cc: <a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" data-auth="NotApplicable">petsc-users@mcs.anl.gov</a>><br>
> > Subject: Re: [petsc-users] Slepc, shell matrix, parallel, halo exchange<br>
> > <br>
> > On Wed, Sep 21, 2022 at 7:41 AM feng wang <<a href="mailto:snailsoar@hotmail.com" data-auth="NotApplicable">snailsoar@hotmail.com</a>> wrote:<br>
> > Hello,<br>
> > <br>
> > I am using Slepc with a shell matrix. The sequential version seems working and now I am trying to make it run in parallel.<br>
> > <br>
> > The partition of the domain is done, I am not sure how to do the halo exchange in the shell matrix in Slepc. I have a parallel version of matrix-free GMRES in my code with Petsc. I was using VecCreateGhostBlock to create vector with ghost cells, and then
used VecGhostUpdateBegin/End for the halo exchange in the shell matrix, would this be the same for Slepc?<br>
> > <br>
> > That will be enough for the MatMult(). You would also have to use a SLEPc EPS that only needed MatMult().<br>
> > <br>
> > Thanks,<br>
> > <br>
> > Matt<br>
> > <br>
> > Thanks,<br>
> > Feng <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > -- <br>
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> > -- Norbert Wiener<br>
> > <br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable">https://www.cse.buffalo.edu/~knepley/</a><br>
<br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" data-auth="NotApplicable">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>