<div dir="ltr"><div dir="ltr">On Tue, Apr 12, 2022 at 7:02 AM Berend van Wachem <<a href="mailto:berend.vanwachem@ovgu.de">berend.vanwachem@ovgu.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear Matt,<br>
<br>
In our code, the size of the overlap is determined in runtime, based on <br>
some calculations. Therefore, we cannot specify it using the <br>
-dm_distribute_overlap option.<br>
<br>
Is there an alternative to set the overlap from within the code before <br>
calling DMPlexDistribute() from the code?</blockquote><div><br></div><div>You could call PetscOptionsSetValue()</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Or is the best way to call <br>
DMPlexDistributeSetDefault(*NewDM, PETSC_FALSE), and call <br>
DMPlexDistribute() with the required value of the overlap?<br></blockquote><div><br></div><div>Yes, for custom things, turning off the automatic stuff might be the best option.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks, best, Berend.<br>
<br>
<br>
<br>
On 4/12/22 12:49, Matthew Knepley wrote:<br>
> On Tue, Apr 12, 2022 at 2:50 AM Berend van Wachem <br>
> <<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>>> wrote:<br>
> <br>
> Dear Matt,<br>
> <br>
> Thank you very much - I can confirm that that works.<br>
> <br>
> I have one question about your remark:<br>
> <br>
> > Also, the call to DMPlexDistribute() here (and the Partitioner<br>
> calls)<br>
> > are now superfluous.<br>
> <br>
> If I shouldn't call DMPlexDistribute(),<br>
> <br>
> <br>
> I just meant that DMPlexDistribute() was being called automatically from <br>
> within DMSetFromOptions()<br>
> <br>
> how should I set the overlap of<br>
> the DM from within the code (our code determines the overlap required)?<br>
> <br>
> <br>
> -dm_distribute_overlap <n> - The size of the overlap halo<br>
> <br>
> from <a href="https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html" rel="noreferrer" target="_blank">https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html</a> <br>
> <<a href="https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html" rel="noreferrer" target="_blank">https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html</a>><br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Many thanks, best regards,<br>
> <br>
> Berend.<br>
> <br>
> <br>
> <br>
> <br>
> On 4/11/22 16:23, Matthew Knepley wrote:<br>
> > On Wed, Apr 6, 2022 at 9:41 AM Berend van Wachem<br>
> > <<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>><br>
> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>>>><br>
> wrote:<br>
> ><br>
> > Dear Matt,<br>
> ><br>
> > I have made a small working example of cloning a DM,<br>
> illustrating the<br>
> > problem we have.<br>
> > In the attached code, I wrote a function<br>
> 'CloneDMWithNewSection', which<br>
> > clones a section and puts a different number of fields on it.<br>
> ><br>
> > The code itself prints the number of local cells of the DM, which<br>
> > changes as the DM is cloned.<br>
> > In our code, we assume that the cloned DM should have exactly<br>
> the same<br>
> > partitioning - this was the behaviour in PETSc versions prior<br>
> to 3.17.<br>
> ><br>
> > If I run the attached code on 2 processors, I get:<br>
> ><br>
> > First DM: Processor 1 reports Start: 0, End 4000 giving<br>
> number of local<br>
> > cells: 4000<br>
> > First DM: Processor 0 reports Start: 0, End 4000 giving<br>
> number of local<br>
> > cells: 4000<br>
> ><br>
> > Cloned DM: Processor 1 reports Start: 0, End 3984 giving<br>
> number of<br>
> > local<br>
> > cells: 3984<br>
> > Cloned DM: Processor 0 reports Start: 0, End 4016 giving<br>
> number of<br>
> > local<br>
> > cells: 4016<br>
> ><br>
> > Maybe we are doing something wrong in the function<br>
> > CloneDMWithNewSection?<br>
> ><br>
> ><br>
> > I apologize for taking so long on this. Jed persuaded me to<br>
> change the<br>
> > default. Now, when you<br>
> > call DMSetFromOptions() it distributes by default, rather than<br>
> requiring<br>
> > you to explicitly call it.<br>
> > You can shut this behavior off, so that if you add<br>
> ><br>
> > ierr = DMPlexDistributeSetDefault(*NewDM,<br>
> PETSC_FALSE);CHKERRQ(ierr);<br>
> ><br>
> > right after DMClone(), you will preserve the layout you have.<br>
> ><br>
> > Also, the call to DMPlexDistribute() here (and the Partitioner<br>
> calls)<br>
> > are now superfluous.<br>
> ><br>
> > Thanks,<br>
> ><br>
> > Matt<br>
> ><br>
> > Many thanks for looking into this, best regards,<br>
> > Berend.<br>
> ><br>
> ><br>
> ><br>
> > On 4/4/22 23:05, Matthew Knepley wrote:<br>
> > > On Mon, Apr 4, 2022 at 3:36 PM Berend van Wachem<br>
> > > <<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a><br>
> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a><br>
> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>>><br>
> > <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a><br>
> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a><br>
> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>>>>><br>
> > wrote:<br>
> > ><br>
> > > Dear Petsc team,<br>
> > ><br>
> > > Since about 2 years we have been using Petsc with<br>
> DMPlex, but<br>
> > since<br>
> > > upgrading our code to Petsc-3.17.0 something has broken.<br>
> > ><br>
> > > First we generate a DM from a DMPlex with<br>
> > DMPlexCreateFromFile or<br>
> > > creating one with DMPlexCreateBoxMesh. Then the DM is<br>
> > distributed with<br>
> > > DMPlexDistribute. This DM works fine and we set a numer of<br>
> > fields and<br>
> > > set a section to it.<br>
> > > However, on the same mesh we also want to solve a<br>
> problem with a<br>
> > > different number of fields, and therefore we create a<br>
> clone<br>
> > of this<br>
> > > original DM, using the code:<br>
> > ><br>
> > > DMClone(OriginalDM, NewDM);<br>
> > > DMClearDS(*NewDM);<br>
> > > PetscCalloc2(1, &NumComp, 4, &NumDof);<br>
> > > NumComp[0] = 1;<br>
> > > NumDof[3] = NFields;<br>
> > > DMSetNumFields(*NewDM, 1);<br>
> > > DMSetFromOptions(*NewDM);<br>
> > > DMPlexCreateSection(*NewDM, NULL, NumComp, NumDof, 0,<br>
> NULL,<br>
> > NULL, NULL,<br>
> > > NULL, §ion);<br>
> > > DMSetLocalSection(*NewDM, section);<br>
> > > PetscFree2(NumComp, NumDof);<br>
> > > PetscSectionDestroy(§ion);<br>
> > ><br>
> > > However, with Petsc-3.17.0, the *NewDM is corrupt -<br>
> When I call<br>
> > > DMGlobalToLocalBegin with a Global and Local vector<br>
> created<br>
> > with this<br>
> > > NewDM, the code crashes. Indeed, the cloned DM seems to be<br>
> > partitioned<br>
> > > differently than the original DM, as it these two DMs<br>
> have a<br>
> > different<br>
> > > number of local cells.<br>
> > ><br>
> > ><br>
> > > The cloned DM will have exactly the same topology and<br>
> > distribution. This<br>
> > > must be a misinterpretation<br>
> > > of what is happening. We can do a few things:<br>
> > ><br>
> > > 1) Make a small example to show what you are talking about<br>
> > ><br>
> > > 2) Look at a PETSc example that does something similar<br>
> > ><br>
> > > 3) Look directly at your code if I can somehow run it here<br>
> > ><br>
> > > 4) Start doing diagnostics on your code to see what is<br>
> going on<br>
> > ><br>
> > > Which one do you prefer?<br>
> > ><br>
> > > This worked fine in Petsc releases before 3.17 (e.g.<br>
> 3.16.5).<br>
> > So my<br>
> > > question is: what has changed? Am I doing something wrong,<br>
> > which should<br>
> > > be changed for using with Petsc-3.17?<br>
> > ><br>
> > ><br>
> > > I don't think any of this should have changed, so this<br>
> should be<br>
> > > something simple.<br>
> > ><br>
> > > Thanks,<br>
> > ><br>
> > > Matt<br>
> > ><br>
> > > Thanks, best regards,<br>
> > ><br>
> > > Berend.<br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > What most experimenters take for granted before they begin<br>
> their<br>
> > > experiments is infinitely more interesting than any<br>
> results to which<br>
> > > their experiments lead.<br>
> > > -- Norbert Wiener<br>
> > ><br>
> > > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
> > <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > What most experimenters take for granted before they begin their<br>
> > experiments is infinitely more interesting than any results to which<br>
> > their experiments lead.<br>
> > -- Norbert Wiener<br>
> ><br>
> > <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a>><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a><br>
> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>>><br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>