[petsc-users] Interpreting Redistribution SF

[Matthew Knepley]

On Thu, Jan 19, 2023 at 10:12 PM Nicholas Arnold-Medabalimi <
narnoldm at umich.edu> wrote:

> Ok thanks for the clarification. In theory, if before the Reduction back
> to the original distribution, if I call DMGlobaltoLocal then even with
> MPI_REPLACE all the leafs corresponding to the original root should have
> the same value so I won't have an ambiguity, correct?
>

That is right, so it should give you the result you expect.

  Thanks,

     Matt


> On Thu, Jan 19, 2023 at 9:28 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Jan 19, 2023 at 9:13 PM Nicholas Arnold-Medabalimi <
>> narnoldm at umich.edu> wrote:
>>
>>> Hi Matt
>>>
>>> Yep, that makes sense and is consistent.
>>>
>>> My question is a little more specific. So let's say I take an
>>> initial mesh and distribute it and get the distribution SF with an overlap
>>> of one. Consider a cell that is a root on process 0 and a leaf on process 1
>>> after the distribution.
>>>
>>> Will the distribution pointSF have an entry for the cell that is a leaf
>>> in the ghost cell sense?
>>>
>>> I guess, in short does the distribution SF only have entries for the
>>> movement of points that are roots in the ghost SF?
>>>
>>
>> I do not understand the question. Suppose that a certain cell, say 0, in
>> the original distribution goes to two different processes, say 0 and 1, and
>> will happen when you distribute with overlap. Then the migration SF has two
>> leaf entries for that cell, one from process 0 and one from process 1. They
>> both point to root cell 0 on process 0.
>>
>>
>>> Sorry if this is a little unclear.
>>>
>>> Maybe my usage will be a bit clearer. I am generating a distributionSF
>>> (type 2 in your desc) then using that to generate a dof distribution(type
>>> 3) using the section information. I then pass the information from the
>>> initial distribution to new distribution with PetscSFBcast with
>>> MPI_REPLACE. That scatters the vector to the new distribution. I then do
>>> "stuff" and now want to redistribute back. So I pass the same dof
>>> distributionSF but call PetscSFReduce with MPI_REPLACE. My concern is I am
>>> only setting the root cell values on each partition. So if the ghost cells
>>> are part of the distribution SF there will be multiple cells reducing to
>>> the original distribution cell?
>>>
>>
>> Yes, definitely.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks
>>> Nicholas
>>>
>>>
>>> On Thu, Jan 19, 2023 at 8:28 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jan 19, 2023 at 11:58 AM Nicholas Arnold-Medabalimi <
>>>> narnoldm at umich.edu> wrote:
>>>>
>>>>> Hi Petsc Users
>>>>>
>>>>> I'm working with a distribution start forest generated by
>>>>> DMPlexDistribute and PetscSFBcast and Reduce to move data between the
>>>>> initial distribution and the distribution generated by DMPlex Distribute.
>>>>>
>>>>> I'm trying to debug some values that aren't being copied properly and
>>>>> wanted to verify I understand how a redistribution SF works compared with a
>>>>> SF that describes overlapped points.
>>>>>
>>>>>   [0] 0 <- (0,7)     point 0 on the distributed plex is point 7 on
>>>>> process 0 on the initial distribution
>>>>>   [0] 1 <- (0,8)     point 1 on the distributed plex is point 8 on
>>>>> process 0 on the initial distribution
>>>>>   [0] 2 <- (0,9)
>>>>>   [0] 3 <- (0,10)
>>>>>   [0] 4 <- (0,11)
>>>>>
>>>>>   [1] 0 <- (1,0)     point 0 on the distributed plex is point 0 on
>>>>> process 1 on the initial distribution
>>>>>   [1] 1 <- (1,1)
>>>>>   [1] 2 <- (1,2)
>>>>>   [1] 3 <- (0,0)     point 3 on the distributed plex is point 0 on
>>>>> process 0 on the initial distribution
>>>>>   [1] 4 <- (0,1)
>>>>>   [1] 5 <- (0,2)
>>>>>
>>>>>  my confusion I think is how does the distributionSF inform of what
>>>>> cells will be leafs on the distribution?
>>>>>
>>>>
>>>> I should eventually write something to clarify this. I am using SF in
>>>> (at least) two different ways.
>>>>
>>>> First, there is a familiar SF that we use for dealing with "ghost"
>>>> points. These are replicated points where one process
>>>> is said to "own" the point and another process is said to hold a
>>>> "ghost". The ghost points are leaves in the SF which
>>>> point back to the root point owned by another process. We call this the
>>>> pointSF for a DM.
>>>>
>>>> Second, we have a migration SF. Here the root points give the original
>>>> point distribution. The leaf points give the new
>>>> point distribution. Thus a PetscSFBcast() pushes points from the
>>>> original to new distribution, which is what we mean
>>>> by a migration.
>>>>
>>>> Third, instead of point values, we might want to communicate fields
>>>> over those points. For this we make new SFes,
>>>> where the numbering does not refer to points, but rather to dofs.
>>>>
>>>> Does this make sense?
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Sincerely
>>>>> Nicholas
>>>>>
>>>>> --
>>>>> Nicholas Arnold-Medabalimi
>>>>>
>>>>> Ph.D. Candidate
>>>>> Computational Aeroscience Lab
>>>>> University of Michigan
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> Nicholas Arnold-Medabalimi
>>>
>>> Ph.D. Candidate
>>> Computational Aeroscience Lab
>>> University of Michigan
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Nicholas Arnold-Medabalimi
>
> Ph.D. Candidate
> Computational Aeroscience Lab
> University of Michigan
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20230120/49cba34e/attachment-0001.html>