[petsc-users] Interpreting Redistribution SF

Thu Jan 19 21:12:19 CST 2023

Ok thanks for the clarification. In theory, if before the Reduction back to
the original distribution, if I call DMGlobaltoLocal then even with
MPI_REPLACE all the leafs corresponding to the original root should have
the same value so I won't have an ambiguity, correct?

On Thu, Jan 19, 2023 at 9:28 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi <
> narnoldm at umich.edu> wrote:
>
>> Hi Matt
>>
>> Yep, that makes sense and is consistent.
>>
>> My question is a little more specific. So let's say I take an
>> initial mesh and distribute it and get the distribution SF with an overlap
>> of one. Consider a cell that is a root on process 0 and a leaf on process 1
>> after the distribution.
>>
>> Will the distribution pointSF have an entry for the cell that is a leaf
>> in the ghost cell sense?
>>
>> I guess, in short does the distribution SF only have entries for the
>> movement of points that are roots in the ghost SF?
>>
>
> I do not understand the question. Suppose that a certain cell, say 0, in
> the original distribution goes to two different processes, say 0 and 1, and
> will happen when you distribute with overlap. Then the migration SF has two
> leaf entries for that cell, one from process 0 and one from process 1. They
> both point to root cell 0 on process 0.
>
>
>> Sorry if this is a little unclear.
>>
>> Maybe my usage will be a bit clearer. I am generating a distributionSF
>> (type 2 in your desc) then using that to generate a dof distribution(type
>> 3) using the section information. I then pass the information from the
>> initial distribution to new distribution with PetscSFBcast with
>> MPI_REPLACE. That scatters the vector to the new distribution. I then do
>> "stuff" and now want to redistribute back. So I pass the same dof
>> distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am
>> only setting the root cell values on each partition. So if the ghost cells
>> are part of the distribution SF there will be multiple cells reducing to
>> the original distribution cell?
>>
>
> Yes, definitely.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks
>> Nicholas
>>
>>
>> On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi <
>>> narnoldm at umich.edu> wrote:
>>>
>>>> Hi Petsc Users
>>>>
>>>> I'm working with a distribution start forest generated by
>>>> DMPlexDistribute and PetscSFBcast and Reduce to move data between the
>>>> initial distribution and the distribution generated by DMPlex Distribute.
>>>>
>>>> I'm trying to debug some values that aren't being copied properly and
>>>> wanted to verify I understand how a redistribution SF works compared with a
>>>> SF that describes overlapped points.
>>>>
>>>>   [0] 0 <- (0,7)     point 0 on the distributed plex is point 7 on
>>>> process 0 on the initial distribution
>>>>   [0] 1 <- (0,8)     point 1 on the distributed plex is point 8 on
>>>> process 0 on the initial distribution
>>>>   [0] 2 <- (0,9)
>>>>   [0] 3 <- (0,10)
>>>>   [0] 4 <- (0,11)
>>>>
>>>>   [1] 0 <- (1,0)     point 0 on the distributed plex is point 0 on
>>>> process 1 on the initial distribution
>>>>   [1] 1 <- (1,1)
>>>>   [1] 2 <- (1,2)
>>>>   [1] 3 <- (0,0)     point 3 on the distributed plex is point 0 on
>>>> process 0 on the initial distribution
>>>>   [1] 4 <- (0,1)
>>>>   [1] 5 <- (0,2)
>>>>
>>>>  my confusion I think is how does the distributionSF inform of what
>>>> cells will be leafs on the distribution?
>>>>
>>>
>>> I should eventually write something to clarify this. I am using SF in
>>> (at least) two different ways.
>>>
>>> First, there is a familiar SF that we use for dealing with "ghost"
>>> points. These are replicated points where one process
>>> is said to "own" the point and another process is said to hold a
>>> "ghost". The ghost points are leaves in the SF which
>>> point back to the root point owned by another process. We call this the
>>> pointSF for a DM.
>>>
>>> Second, we have a migration SF. Here the root points give the original
>>> point distribution. The leaf points give the new
>>> point distribution. Thus a PetscSFBcast() pushes points from the
>>> original to new distribution, which is what we mean
>>> by a migration.
>>>
>>> Third, instead of point values, we might want to communicate fields over
>>> those points. For this we make new SFes,
>>> where the numbering does not refer to points, but rather to dofs.
>>>
>>> Does this make sense?
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Sincerely
>>>> Nicholas
>>>>
>>>> --
>>>> Nicholas Arnold-Medabalimi
>>>>
>>>> Ph.D. Candidate
>>>> Computational Aeroscience Lab
>>>> University of Michigan
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Nicholas Arnold-Medabalimi
>>
>> Ph.D. Candidate
>> Computational Aeroscience Lab
>> University of Michigan
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>

-- 
Nicholas Arnold-Medabalimi

Ph.D. Candidate
Computational Aeroscience Lab
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230119/626e38bd/attachment.html>