<div dir="ltr"><div dir="ltr">On Wed, Apr 6, 2022 at 9:41 AM Berend van Wachem <<a href="mailto:berend.vanwachem@ovgu.de">berend.vanwachem@ovgu.de</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear Matt,<br>
<br>
I have made a small working example of cloning a DM, illustrating the <br>
problem we have.<br>
In the attached code, I wrote a function 'CloneDMWithNewSection', which <br>
clones a section and puts a different number of fields on it.<br>
<br>
The code itself prints the number of local cells of the DM, which <br>
changes as the DM is cloned.<br>
In our code, we assume that the cloned DM should have exactly the same <br>
partitioning - this was the behaviour in PETSc versions prior to 3.17.<br>
<br>
If I run the attached code on 2 processors, I get:<br>
<br>
First DM: Processor 1 reports Start: 0, End 4000 giving number of local <br>
cells: 4000<br>
First DM: Processor 0 reports Start: 0, End 4000 giving number of local <br>
cells: 4000<br>
<br>
Cloned DM: Processor 1 reports Start: 0, End 3984 giving number of local <br>
cells: 3984<br>
Cloned DM: Processor 0 reports Start: 0, End 4016 giving number of local <br>
cells: 4016<br>
<br>
Maybe we are doing something wrong in the function CloneDMWithNewSection?<br></blockquote><div><br></div><div>I apologize for taking so long on this. Jed persuaded me to change the default. Now, when you</div><div>call DMSetFromOptions() it distributes by default, rather than requiring you to explicitly call it.</div><div>You can shut this behavior off, so that if you add</div><div><br></div><div> ierr = DMPlexDistributeSetDefault(*NewDM, PETSC_FALSE);CHKERRQ(ierr);<br></div><div><br></div><div>right after DMClone(), you will preserve the layout you have.</div><div><br></div><div>Also, the call to DMPlexDistribute() here (and the Partitioner calls) are now superfluous.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Many thanks for looking into this, best regards,<br>
Berend.<br>
<br>
<br>
<br>
On 4/4/22 23:05, Matthew Knepley wrote:<br>
> On Mon, Apr 4, 2022 at 3:36 PM Berend van Wachem <br>
> <<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a> <mailto:<a href="mailto:berend.vanwachem@ovgu.de" target="_blank">berend.vanwachem@ovgu.de</a>>> wrote:<br>
> <br>
> Dear Petsc team,<br>
> <br>
> Since about 2 years we have been using Petsc with DMPlex, but since<br>
> upgrading our code to Petsc-3.17.0 something has broken.<br>
> <br>
> First we generate a DM from a DMPlex with DMPlexCreateFromFile or<br>
> creating one with DMPlexCreateBoxMesh. Then the DM is distributed with<br>
> DMPlexDistribute. This DM works fine and we set a numer of fields and<br>
> set a section to it.<br>
> However, on the same mesh we also want to solve a problem with a<br>
> different number of fields, and therefore we create a clone of this<br>
> original DM, using the code:<br>
> <br>
> DMClone(OriginalDM, NewDM);<br>
> DMClearDS(*NewDM);<br>
> PetscCalloc2(1, &NumComp, 4, &NumDof);<br>
> NumComp[0] = 1;<br>
> NumDof[3] = NFields;<br>
> DMSetNumFields(*NewDM, 1);<br>
> DMSetFromOptions(*NewDM);<br>
> DMPlexCreateSection(*NewDM, NULL, NumComp, NumDof, 0, NULL, NULL, NULL,<br>
> NULL, §ion);<br>
> DMSetLocalSection(*NewDM, section);<br>
> PetscFree2(NumComp, NumDof);<br>
> PetscSectionDestroy(§ion);<br>
> <br>
> However, with Petsc-3.17.0, the *NewDM is corrupt - When I call<br>
> DMGlobalToLocalBegin with a Global and Local vector created with this<br>
> NewDM, the code crashes. Indeed, the cloned DM seems to be partitioned<br>
> differently than the original DM, as it these two DMs have a different<br>
> number of local cells.<br>
> <br>
> <br>
> The cloned DM will have exactly the same topology and distribution. This <br>
> must be a misinterpretation<br>
> of what is happening. We can do a few things:<br>
> <br>
> 1) Make a small example to show what you are talking about<br>
> <br>
> 2) Look at a PETSc example that does something similar<br>
> <br>
> 3) Look directly at your code if I can somehow run it here<br>
> <br>
> 4) Start doing diagnostics on your code to see what is going on<br>
> <br>
> Which one do you prefer?<br>
> <br>
> This worked fine in Petsc releases before 3.17 (e.g. 3.16.5). So my<br>
> question is: what has changed? Am I doing something wrong, which should<br>
> be changed for using with Petsc-3.17?<br>
> <br>
> <br>
> I don't think any of this should have changed, so this should be <br>
> something simple.<br>
> <br>
> Thanks,<br>
> <br>
> Matt<br>
> <br>
> Thanks, best regards,<br>
> <br>
> Berend.<br>
> <br>
> <br>
> <br>
> -- <br>
> What most experimenters take for granted before they begin their <br>
> experiments is infinitely more interesting than any results to which <br>
> their experiments lead.<br>
> -- Norbert Wiener<br>
> <br>
> <a href="https://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">https://www.cse.buffalo.edu/~knepley/</a> <<a href="http://www.cse.buffalo.edu/~knepley/" rel="noreferrer" target="_blank">http://www.cse.buffalo.edu/~knepley/</a>></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>