[petsc-users] Interpreting Redistribution SF

[Matthew Knepley]

On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Hi Matt
>
> Yep, that makes sense and is consistent.
>
> My question is a little more specific. So let's say I take an initial mesh
> and distribute it and get the distribution SF with an overlap of one.
> Consider a cell that is a root on process 0 and a leaf on process 1 after
> the distribution.
>
> Will the distribution pointSF have an entry for the cell that is a leaf in
> the ghost cell sense?
>
> I guess, in short does the distribution SF only have entries for the
> movement of points that are roots in the ghost SF?
>

I do not understand the question. Suppose that a certain cell, say 0, in
the original distribution goes to two different processes, say 0 and 1, and
will happen when you distribute with overlap. Then the migration SF has two
leaf entries for that cell, one from process 0 and one from process 1. They
both point to root cell 0 on process 0.


> Sorry if this is a little unclear.
>
> Maybe my usage will be a bit clearer. I am generating a distributionSF
> (type 2 in your desc) then using that to generate a dof distribution(type
> 3) using the section information. I then pass the information from the
> initial distribution to new distribution with PetscSFBcast with
> MPI_REPLACE. That scatters the vector to the new distribution. I then do
> "stuff" and now want to redistribute back. So I pass the same dof
> distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am
> only setting the root cell values on each partition. So if the ghost cells
> are part of the distribution SF there will be multiple cells reducing to
> the original distribution cell?
>

Yes, definitely.

  Thanks,

     Matt


> Thanks
> Nicholas
>
>
> On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi <
>> narnoldm at umich.edu> wrote:
>>
>>> Hi Petsc Users
>>>
>>> I'm working with a distribution start forest generated by
>>> DMPlexDistribute and PetscSFBcast and Reduce to move data between the
>>> initial distribution and the distribution generated by DMPlex Distribute.
>>>
>>> I'm trying to debug some values that aren't being copied properly and
>>> wanted to verify I understand how a redistribution SF works compared with a
>>> SF that describes overlapped points.
>>>
>>>   [0] 0 <- (0,7)     point 0 on the distributed plex is point 7 on
>>> process 0 on the initial distribution
>>>   [0] 1 <- (0,8)     point 1 on the distributed plex is point 8 on
>>> process 0 on the initial distribution
>>>   [0] 2 <- (0,9)
>>>   [0] 3 <- (0,10)
>>>   [0] 4 <- (0,11)
>>>
>>>   [1] 0 <- (1,0)     point 0 on the distributed plex is point 0 on
>>> process 1 on the initial distribution
>>>   [1] 1 <- (1,1)
>>>   [1] 2 <- (1,2)
>>>   [1] 3 <- (0,0)     point 3 on the distributed plex is point 0 on
>>> process 0 on the initial distribution
>>>   [1] 4 <- (0,1)
>>>   [1] 5 <- (0,2)
>>>
>>>  my confusion I think is how does the distributionSF inform of what
>>> cells will be leafs on the distribution?
>>>
>>
>> I should eventually write something to clarify this. I am using SF in (at
>> least) two different ways.
>>
>> First, there is a familiar SF that we use for dealing with "ghost"
>> points. These are replicated points where one process
>> is said to "own" the point and another process is said to hold a "ghost".
>> The ghost points are leaves in the SF which
>> point back to the root point owned by another process. We call this the
>> pointSF for a DM.
>>
>> Second, we have a migration SF. Here the root points give the original
>> point distribution. The leaf points give the new
>> point distribution. Thus a PetscSFBcast() pushes points from the original
>> to new distribution, which is what we mean
>> by a migration.
>>
>> Third, instead of point values, we might want to communicate fields over
>> those points. For this we make new SFes,
>> where the numbering does not refer to points, but rather to dofs.
>>
>> Does this make sense?
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Sincerely
>>> Nicholas
>>>
>>> --
>>> Nicholas Arnold-Medabalimi
>>>
>>> Ph.D. Candidate
>>> Computational Aeroscience Lab
>>> University of Michigan
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230119/955a3b30/attachment-0001.html>