<div dir="ltr"><div dir="ltr">On Mon, May 25, 2020 at 11:19 AM Lucas Banting <<a href="mailto:bantingl@myumanitoba.ca">bantingl@myumanitoba.ca</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
So DM_BOUNDARY_GHOSTED values are not updated then in the DMDALocalToLocal routines. I should instead use DM_BOUNDARY_NONE and make the domain larger by 2 elements if I need those values to be shared between processes then? Is that the best approach?</div></div></div></blockquote><div><br></div><div>I think we need to clarify what PETSc means by these things.</div><div><br></div><div>We divide the original grid into non-overlapping pieces. These are the vertices that are ordered for the global numbering,</div><div>and those that we own are called the "local" vertices. In order to calculate the value at a local vertex, we need information</div><div>from neighboring vertices. Sometimes these are outside the portion we own. We call these "ghost" vertices. Thus</div><div><br></div><div> Global Numbering --> local vertices</div><div> Local Numbering --> local+ghost vertices</div><div><br></div><div>At a boundary, we have to decide how our stencil behaves. If there is just nothing there, we would have type NONE. Some stencil</div><div>indices for a boundary vertex are invalid. If instead we decide that the stencil should wrap around the domain, indexing into vertices</div><div>owned by another process, that is type PERIODIC. If the stencil wraps back onto this process, it is type MIRROR. If the stencil</div><div>indexes into some storage space in which I can put boundary values, it is type GHOSTED.</div><div><br></div><div>Now if we do LocalToGlobal. We are copying unknowns from Global storage to Local storage. This is just 1-to-1 if we have NONE,</div><div>since no extra unknowns were added. For PERIODIC or MIRROR, the extra local unknowns are mapped to other global unknowns,</div><div>and thus we have a copy, possibly with communication. For GHOSTED, the extra local unknowns are just storage and do not</div><div>correspond to other global unknowns. Thus there is no copy.</div><div><br></div><div>We can think of LocalToLocal as just LocalToGlobal followed by GlobalToLocal but done in one step. Since GHOSTED is not affected</div><div>by LocalToGlobal, nothing happens. I am not sure what you are trying to achieve with the boundary condition.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
Lucas</div>
<br>
</div>
<div id="gmail-m_6514592758445884529appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_6514592758445884529divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Sent:</b> Friday, May 22, 2020 8:03 PM<br>
<b>To:</b> Lucas Banting <<a href="mailto:bantingl@myumanitoba.ca" target="_blank">bantingl@myumanitoba.ca</a>><br>
<b>Cc:</b> PETSc <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] Question about DMLocalToLocal for DM_BOUNDARY_GHOSTED conditions</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Fri, May 22, 2020 at 4:34 PM Lucas Banting <<a href="mailto:bantingl@myumanitoba.ca" target="_blank">bantingl@myumanitoba.ca</a>> wrote:<br>
</div>
<div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am converting a serial code to parallel in fortran with petsc. I am using the DMDA to manage communication of the information that used to be in old two-dimensional fortran arrays.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I noticed when using DMLocalToLocalBegin/End, not all the ghost values in the array at the DM_BOUNDARY_GHOSTED area is updated. Is this expected behaviour?</div>
</div>
</blockquote>
<div><br>
</div>
<div>I believe so. GHOSTED is user managed space. We do not touch it.</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I read through this thread: <a href="https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-May/029252.html" id="gmail-m_6514592758445884529x_gmail-m_-7081108383524483544LPlnk791893" target="_blank">
https://lists.mcs.anl.gov/mailman/htdig/petsc-users/2016-May/029252.html</a> and saw someone had a similar question, but the answer was not clear to me.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
If this is expected behaviour, how should I instead update these values in my arrays? I was using DM_BOUNDARY_GHOSTED as I needed the extra ghost cells for some subroutines, but I do not need them in my matrix from DMCreateMatrix.</div>
</div>
</blockquote>
<div><br>
</div>
<div>You fill them in the local vector.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I am using Petsc 3.12.4 and open MPI 3.1.4.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Lucas Banting<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>