[petsc-users] DMCloning from a DMPlex has changed in Petsc-3.17.0?
Matthew Knepley
knepley at gmail.com
Tue Apr 12 06:18:15 CDT 2022
On Tue, Apr 12, 2022 at 7:02 AM Berend van Wachem <berend.vanwachem at ovgu.de>
wrote:
> Dear Matt,
>
> In our code, the size of the overlap is determined in runtime, based on
> some calculations. Therefore, we cannot specify it using the
> -dm_distribute_overlap option.
>
> Is there an alternative to set the overlap from within the code before
> calling DMPlexDistribute() from the code?
You could call PetscOptionsSetValue()
> Or is the best way to call
> DMPlexDistributeSetDefault(*NewDM, PETSC_FALSE), and call
> DMPlexDistribute() with the required value of the overlap?
>
Yes, for custom things, turning off the automatic stuff might be the best
option.
Thanks,
Matt
> Thanks, best, Berend.
>
>
>
> On 4/12/22 12:49, Matthew Knepley wrote:
> > On Tue, Apr 12, 2022 at 2:50 AM Berend van Wachem
> > <berend.vanwachem at ovgu.de <mailto:berend.vanwachem at ovgu.de>> wrote:
> >
> > Dear Matt,
> >
> > Thank you very much - I can confirm that that works.
> >
> > I have one question about your remark:
> >
> > > Also, the call to DMPlexDistribute() here (and the Partitioner
> > calls)
> > > are now superfluous.
> >
> > If I shouldn't call DMPlexDistribute(),
> >
> >
> > I just meant that DMPlexDistribute() was being called automatically from
> > within DMSetFromOptions()
> >
> > how should I set the overlap of
> > the DM from within the code (our code determines the overlap
> required)?
> >
> >
> > -dm_distribute_overlap <n> - The size of the overlap halo
> >
> > from https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html
> > <https://petsc.org/main/docs/manualpages/DM/DMSetFromOptions.html>
> >
> > Thanks,
> >
> > Matt
> >
> > Many thanks, best regards,
> >
> > Berend.
> >
> >
> >
> >
> > On 4/11/22 16:23, Matthew Knepley wrote:
> > > On Wed, Apr 6, 2022 at 9:41 AM Berend van Wachem
> > > <berend.vanwachem at ovgu.de <mailto:berend.vanwachem at ovgu.de>
> > <mailto:berend.vanwachem at ovgu.de <mailto:berend.vanwachem at ovgu.de>>>
> > wrote:
> > >
> > > Dear Matt,
> > >
> > > I have made a small working example of cloning a DM,
> > illustrating the
> > > problem we have.
> > > In the attached code, I wrote a function
> > 'CloneDMWithNewSection', which
> > > clones a section and puts a different number of fields on it.
> > >
> > > The code itself prints the number of local cells of the DM,
> which
> > > changes as the DM is cloned.
> > > In our code, we assume that the cloned DM should have exactly
> > the same
> > > partitioning - this was the behaviour in PETSc versions prior
> > to 3.17.
> > >
> > > If I run the attached code on 2 processors, I get:
> > >
> > > First DM: Processor 1 reports Start: 0, End 4000 giving
> > number of local
> > > cells: 4000
> > > First DM: Processor 0 reports Start: 0, End 4000 giving
> > number of local
> > > cells: 4000
> > >
> > > Cloned DM: Processor 1 reports Start: 0, End 3984 giving
> > number of
> > > local
> > > cells: 3984
> > > Cloned DM: Processor 0 reports Start: 0, End 4016 giving
> > number of
> > > local
> > > cells: 4016
> > >
> > > Maybe we are doing something wrong in the function
> > > CloneDMWithNewSection?
> > >
> > >
> > > I apologize for taking so long on this. Jed persuaded me to
> > change the
> > > default. Now, when you
> > > call DMSetFromOptions() it distributes by default, rather than
> > requiring
> > > you to explicitly call it.
> > > You can shut this behavior off, so that if you add
> > >
> > > ierr = DMPlexDistributeSetDefault(*NewDM,
> > PETSC_FALSE);CHKERRQ(ierr);
> > >
> > > right after DMClone(), you will preserve the layout you have.
> > >
> > > Also, the call to DMPlexDistribute() here (and the Partitioner
> > calls)
> > > are now superfluous.
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > > Many thanks for looking into this, best regards,
> > > Berend.
> > >
> > >
> > >
> > > On 4/4/22 23:05, Matthew Knepley wrote:
> > > > On Mon, Apr 4, 2022 at 3:36 PM Berend van Wachem
> > > > <berend.vanwachem at ovgu.de
> > <mailto:berend.vanwachem at ovgu.de> <mailto:berend.vanwachem at ovgu.de
> > <mailto:berend.vanwachem at ovgu.de>>
> > > <mailto:berend.vanwachem at ovgu.de
> > <mailto:berend.vanwachem at ovgu.de> <mailto:berend.vanwachem at ovgu.de
> > <mailto:berend.vanwachem at ovgu.de>>>>
> > > wrote:
> > > >
> > > > Dear Petsc team,
> > > >
> > > > Since about 2 years we have been using Petsc with
> > DMPlex, but
> > > since
> > > > upgrading our code to Petsc-3.17.0 something has
> broken.
> > > >
> > > > First we generate a DM from a DMPlex with
> > > DMPlexCreateFromFile or
> > > > creating one with DMPlexCreateBoxMesh. Then the DM is
> > > distributed with
> > > > DMPlexDistribute. This DM works fine and we set a
> numer of
> > > fields and
> > > > set a section to it.
> > > > However, on the same mesh we also want to solve a
> > problem with a
> > > > different number of fields, and therefore we create a
> > clone
> > > of this
> > > > original DM, using the code:
> > > >
> > > > DMClone(OriginalDM, NewDM);
> > > > DMClearDS(*NewDM);
> > > > PetscCalloc2(1, &NumComp, 4, &NumDof);
> > > > NumComp[0] = 1;
> > > > NumDof[3] = NFields;
> > > > DMSetNumFields(*NewDM, 1);
> > > > DMSetFromOptions(*NewDM);
> > > > DMPlexCreateSection(*NewDM, NULL, NumComp, NumDof, 0,
> > NULL,
> > > NULL, NULL,
> > > > NULL, §ion);
> > > > DMSetLocalSection(*NewDM, section);
> > > > PetscFree2(NumComp, NumDof);
> > > > PetscSectionDestroy(§ion);
> > > >
> > > > However, with Petsc-3.17.0, the *NewDM is corrupt -
> > When I call
> > > > DMGlobalToLocalBegin with a Global and Local vector
> > created
> > > with this
> > > > NewDM, the code crashes. Indeed, the cloned DM seems
> to be
> > > partitioned
> > > > differently than the original DM, as it these two DMs
> > have a
> > > different
> > > > number of local cells.
> > > >
> > > >
> > > > The cloned DM will have exactly the same topology and
> > > distribution. This
> > > > must be a misinterpretation
> > > > of what is happening. We can do a few things:
> > > >
> > > > 1) Make a small example to show what you are talking about
> > > >
> > > > 2) Look at a PETSc example that does something similar
> > > >
> > > > 3) Look directly at your code if I can somehow run it here
> > > >
> > > > 4) Start doing diagnostics on your code to see what is
> > going on
> > > >
> > > > Which one do you prefer?
> > > >
> > > > This worked fine in Petsc releases before 3.17 (e.g.
> > 3.16.5).
> > > So my
> > > > question is: what has changed? Am I doing something
> wrong,
> > > which should
> > > > be changed for using with Petsc-3.17?
> > > >
> > > >
> > > > I don't think any of this should have changed, so this
> > should be
> > > > something simple.
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > > >
> > > > Thanks, best regards,
> > > >
> > > > Berend.
> > > >
> > > >
> > > >
> > > > --
> > > > What most experimenters take for granted before they begin
> > their
> > > > experiments is infinitely more interesting than any
> > results to which
> > > > their experiments lead.
> > > > -- Norbert Wiener
> > > >
> > > > https://www.cse.buffalo.edu/~knepley/
> > <https://www.cse.buffalo.edu/~knepley/>
> > > <https://www.cse.buffalo.edu/~knepley/
> > <https://www.cse.buffalo.edu/~knepley/>>
> > > <http://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/~knepley/>
> > > <http://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/~knepley/>>>
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to
> which
> > > their experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/
> > <https://www.cse.buffalo.edu/~knepley/>
> > <http://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/~knepley/>>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/ <
> http://www.cse.buffalo.edu/~knepley/>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220412/46709a1a/attachment.html>
More information about the petsc-users
mailing list