[petsc-dev] Petsc "make test" have more failures for --with-openmp=1

[Jed Brown]

Matthew Knepley <knepley at gmail.com> writes:

> On Fri, Mar 19, 2021 at 2:21 PM Jed Brown <jed at jedbrown.org> wrote:
>
>> Matthew Knepley <knepley at gmail.com> writes:
>>
>> >> Notice how the permutations are contained within the vertices {0, ...,
>> 8},
>> >> edges {9, ..., 24}, and cells {25, ..., 32}. I would like to get rid of
>> >> that restriction, but you've said it would have significant non-local
>> >> consequences so I haven't tried.
>> >>
>> >
>> > Yes, that is not a good idea, even in theory. It wrecks the nice
>> contiguity
>> > we use for topological operations.
>> >
>> > As Lawrence points out, this need not affect your smoother because dof
>> > permutations can break this stratification. It has been that way for a
>> > decade.
>>
>> I only want to break dof stratification, not point stratification. The
>> points produced by the stratified RCM are okay (at least as Lawrence
>> described it; not sure how it's done in DMPlexGetOrdering, but can't be
>> anything hard).
>>
>> What code needs to be written to get unstratified RCM-like dofs? Lawrence
>> evidently has this code in Firedrake, but I want it in DMPlex, tested and
>> probably done by default because most users would benefit from it. I could
>> swear you said it would break some other assumptions and thus not
>> everything would work, but perhaps we weren't talking about the same thing.
>>
>
> It is possible that I misunderstood what you were asking for. What Section
> requires for everything to work is that dofs follow the point ordering. 

I think what you mean is that the dofs for any point are contiguous, not that the dofs for point p come before the dofs for point p+1.

> The point ordering can be anything we want since we can use a permutation from the mesh point ordering. However, some stuff depends on all dofs associated with a point being contiguous. Not everything does, since I wrote something for LibMesh which allows them to order in a field major way, instead of point major, but it is probably not possible to individually permute unknowns. Usually, this is not what we want anyway I would assume.

I don't want to break up the dofs for a point, just an unstratified locality-preserving ordering of dofs (not points).

PetscSectionSetUp currently calculates offsets by walking the points in order. I'd like to be able to walk them in a different ordering, perhaps specified via an IS permutation, which could be computed using MATORDERINGRCM or directly via BFS of a closure (saves building a Mat).

1. Would this break anything you can think of?
2. How would you like it implemented?