[petsc-dev] DMPLEX with VecGhost

[Matthew Knepley]

On Thu, Oct 28, 2021 at 10:37 AM Barry Smith <bsmith at petsc.dev> wrote:

>
>
> On Oct 28, 2021, at 10:31 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Thu, Oct 28, 2021 at 9:37 AM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>>   Matt,
>>
>>     How difficult would it be to rework DMPLEX to allow the use of
>> VecGhost? We have performance problems with GPUs with simple DMNETWORK
>> models because the code spends more time uselessly copying the local part
>> of the vector to another vector in global to local and local to global;
>> more than 1/2 the time of the total simulation.
>>
>
> Firedrake already does this because they "vec ghost" their vectors by
> default. Here is what you need:
>
>   When you create the PetscSection, by default it orders the unknowns
> according to the default point numbering. This
>   is what causes the ghost unknowns to be mixed in with the
> local unknowns. However, PetscSection allows you to set
>   a point permutation
>
>
> https://petsc.org/main/docs/manualpages/PetscSection/PetscSectionSetPermutation.html
>
>   This determines the order of dogs by iterating through points in this
> permutation, and you can put all shared points at the end.
>
>
>   How do I know what are shared points to put at the end? Couldn't DMPLEX
> do this automatically with an option? Where is the Firedrake code that does
> this with DMPLEX so I can see it?
>

Yes, it could. I was thinking this as I wrote you, and also thinking how
tired I am :)

Here is how to do it. Suppose you only care about ghost points (Firedrake
wants three layers: totalling internal, roots, and ghosts), then there is a
simple test

  DMGetLocalSection(dm, &s)
  PetscSectionGetChart(s, &pStart, &pEnd);
  DMGetPointSF(dm, &sf);
  PetscSFGetGraph(sf, NULL, &nleaves, &leaves, NULL)
  for (p = pStart; p < pEnd; ++p) {
    PetscFindInt(p, nleaves, leaves, &loc);
    if (loc >= 0) {I am a ghost point}
  }

If you also care about roots (shared points that you own), you also need

  PetscSFComputeDegreeBegin(sf, &rootdegree);
  PetscSFComputeDegreeEnd(sf, &rootdegree);

and then also check

  rootdegree[p] > 0

If you get it working, I will put it inside Plex and make an option to turn
it on.

  Thanks,

     Matt

> Does this make sense?
>
>   Thanks,
>
>      Matt
>
>
>>   Barry
>>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20211028/6a2d3abb/attachment.html>