[petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Matthew Knepley]

On Fri, Mar 19, 2021 at 10:21 AM Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
>
> > On Fri, Mar 19, 2021 at 7:30 AM Lawrence Mitchell <wence at gmx.li> wrote:
> >
> >> > On 19 Mar 2021, at 03:51, Jed Brown <jed at jedbrown.org> wrote:
> >> >
> >> > It's a notable weakness of DMPlex that it does not apply such an
> >> ordering of dofs and I've complained to Matt about it many times over
> the
> >> years, but any blame rests solely with me for not carving out time to
> >> implement it here.
> >>
> >> I think there is a way to do this with plex (at least we do it), like
> so:
> >>
> >> DMPlexGetOrdering(dm, MATORDERINGRCM, NULL, &isperm);
> >>
> >> you might need to then invert this ordering.
> >>
> >> Now when you make a section you either say
> >>
> >> PetscSectionSetPermutation(section, isperm);
> >>
> >
> > You can do that if you only want to permute the dofs, but usually I want
> to
> > reorder everything so that any Section comes out right
> >
> >   DMPlexPermute()
>
> Last we talked, and all that's tested, this can only permute within
> strata, not across strata. That would incur the "basically point Jacobi"
> problem for naive OpenMP smoothers highlighted in Allison's paper and (what
> I care about more) the memory locality impact is similar to moving from
> interlaced fields to segregated fields.
>
> diff --git i/src/dm/impls/plex/tests/ex10.c
> w/src/dm/impls/plex/tests/ex10.c
> index eed9aa878e..cfe35dc1a6 100644
> --- i/src/dm/impls/plex/tests/ex10.c
> +++ w/src/dm/impls/plex/tests/ex10.c
> @@ -80,6 +80,7 @@ PetscErrorCode TestReordering(DM dm, AppCtx *user)
>
>    PetscFunctionBegin;
>    ierr = DMPlexGetOrdering(dm, order, NULL, &perm);CHKERRQ(ierr);
> +  ierr = ISView(perm, NULL);CHKERRQ(ierr);
>    ierr = DMPlexPermute(dm, perm, &pdm);CHKERRQ(ierr);
>    ierr = PetscObjectSetOptionsPrefix((PetscObject) pdm,
> "perm_");CHKERRQ(ierr);
>    ierr = DMSetFromOptions(pdm);CHKERRQ(ierr);
>
>
>
> $ $PETSC_ARCH/tests/dm/impls/plex/tests/ex10 -dim 2 -num_dof 1,1,1
> -perm_dm_view
> IS Object: 1 MPI processes
>   type: general
> Number of indices in set 33
> 0 0
> 1 2
> 2 1
> 3 5
> 4 3
> 5 7
> 6 6
> 7 4
> 8 8
> 9 9
> 10 13
> 11 10
> 12 11
> 13 15
> 14 12
> 15 14
> 16 16
> 17 17
> 18 18
> 19 19
> 20 22
> 21 23
> 22 21
> 23 20
> 24 24
> 25 28
> 26 29
> 27 25
> 28 30
> 29 31
> 30 32
> 31 26
> 32 27
> DM Object: (perm_) 1 MPI processes
>   type: plex
> DM_0x564eba4a45d0_1 in 2 dimensions:
>   0-cells: 9
>   1-cells: 16
>   2-cells: 8
> Labels:
>   celltype: 3 strata with value/size (0 (9), 3 (8), 1 (16))
>   depth: 3 strata with value/size (0 (9), 1 (16), 2 (8))
>   marker: 1 strata with value/size (1 (16))
>   Face Sets: 1 strata with value/size (1 (8))
> Field Field_0:
>   adjacency FEM
> Ordering method rcm reduced bandwidth from 51 to 51
>
>
> Notice how the permutations are contained within the vertices {0, ..., 8},
> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of
> that restriction, but you've said it would have significant non-local
> consequences so I haven't tried.
>

Yes, that is not a good idea, even in theory. It wrecks the nice contiguity
we use for topological operations.

As Lawrence points out, this need not affect your smoother because dof
permutations can break this stratification. It has been that way for a
decade.

   Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210319/770526f2/attachment.html>