[petsc-users] PetscSF Object on Distributed DMPlex for Halo Data Exchange

Mike Michell mi.mike1021 at gmail.com
Fri Jun 3 12:39:43 CDT 2022 

Thanks for the effort. By any chance, is there any rough timeline for that
part can be done?

Thanks,
Mike


> On Tue, May 31, 2022 at 10:26 AM Mike Michell <mi.mike1021 at gmail.com>
> wrote:
>
>> Thank you. But, it seems that PetscSFCreateSectionSF() also requires
>> petscsf.h file. Which header file I should include to call
>> PetscSFCreateSectionSF() from Fortran?
>>
>
> I will have to write a binding. I will send you the MR when I finish.
>
>   THanks,
>
>     Matt
>
>
>> Thanks,
>> Mike
>>
>>
>> On Tue, May 31, 2022 at 10:04 AM Mike Michell <mi.mike1021 at gmail.com>
>>> wrote:
>>>
>>>> As a follow-up question on your example, is it possible to call
>>>> PetscSFCreateRemoteOffsets() from Fortran?
>>>>
>>>> My code is written in .F90 and in "petsc/finclude/" there is no
>>>> petscsf.h so that the code currently cannot find
>>>> PetscSFCreateRemoteOffsets().
>>>>
>>>
>>> I believe if you pass in NULL for remoteOffsets, that function will be
>>> called internally.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>> Mike
>>>>
>>>>
>>>> I will also point out that Toby has created a nice example showing how
>>>>> to create an SF for halo exchange between local vectors.
>>>>>
>>>>>   https://gitlab.com/petsc/petsc/-/merge_requests/5267
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>> On Sun, May 22, 2022 at 9:47 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Sun, May 22, 2022 at 4:28 PM Mike Michell <mi.mike1021 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the reply. The diagram makes sense and is helpful for
>>>>>>> understanding 1D representation.
>>>>>>>
>>>>>>> However, something is still unclear. From your diagram, the number
>>>>>>> of roots per process seems to vary according to run arguments, such as
>>>>>>> "-dm_distribute_overlap", because "the number of roots for a DMPlex is the
>>>>>>> number of mesh points in the local portion of the mesh (cited from your
>>>>>>> answer to my question (1))" will end up change according to that argument.
>>>>>>> However, from my mock-up code, number of roots is independent to
>>>>>>> -dm_distribute_overlap argument. The summation of "number of roots" through
>>>>>>> processes was always equal to number of physical vertex on my mesh, if I
>>>>>>> define the section layout on vertex with 1DOF. But in your diagram example,
>>>>>>> the summation of "nroots" is larger than the actual number of mesh points,
>>>>>>> which is 13.
>>>>>>>
>>>>>>
>>>>>> I do not understand your question. Notice the -dm_distribute_overlap
>>>>>> does _not_ change the owned points for any process. It only puts in new
>>>>>> leaves, so it also never
>>>>>> changes the roots for this way of using the SF.
>>>>>>
>>>>>>
>>>>>>> Also, it is still unclear how to get the size of "roots" from the
>>>>>>> PetscSection & PetscSF on distributed DMPlex?
>>>>>>>
>>>>>>
>>>>>> For an SF mapping ghost dofs in a global vector, the number of roots
>>>>>> is just the size of the local portion of the vector.
>>>>>>
>>>>>>
>>>>>>> In your diagram, how can you tell your code and make it allocate the
>>>>>>> "nroots=7 for P0, nroots=9 for P1, and nroots=7 for P2" arrays before you
>>>>>>> call PetscSFBcastBegin/End()? It seems that we need to define arrays having
>>>>>>> the size of nroots & nleaves before calling PetscSFBcastBegin/End().
>>>>>>>
>>>>>>
>>>>>> I just want to note that this usage is different from the canonical
>>>>>> usage in Plex. It is fine to do this, but this will not match what I do in
>>>>>> the library if you look.
>>>>>> In Plex, I distinguish two linear spaces:
>>>>>>
>>>>>>   1) Global space: This is the vector space for the solvers. Each
>>>>>> point is uniquely represented and owned by some process
>>>>>>
>>>>>>   2) Local space: This is the vector space for assembly. Some points
>>>>>> are represented multiple times.
>>>>>>
>>>>>> I create an SF that maps from the global space (roots) to the local
>>>>>> space (leaves), and it is called in DMGlobalToLocal() (and
>>>>>> associated functions). This
>>>>>> is more natural in FEM. You seem to want an SF that maps between
>>>>>> global vectors. This will also work. The roots would be the local dofs, and
>>>>>> the leaves
>>>>>> would be shared dofs.
>>>>>>
>>>>>>   Does this make sense?
>>>>>>
>>>>>>      Thanks,
>>>>>>
>>>>>>        Matt
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> Here's a diagram of a 1D mesh with overlap and 3 partitions, showing
>>>>>>>> what the petscsf data is for each.  The number of roots is the number of
>>>>>>>> mesh points in the local representation, and the number of leaves is the
>>>>>>>> number of mesh points that are duplicates of mesh points on other
>>>>>>>> processes.  With that in mind, answering your questions
>>>>>>>>
>>>>>>>> > (1) It seems that the "roots" means the number of vertex not
>>>>>>>> considering overlap layer, and "leaves" seems the number of distributed
>>>>>>>> vertex for each processor that includes overlap layer. Can you acknowledge
>>>>>>>> that this is correct understanding? I have tried to find clearer examples
>>>>>>>> from PETSc team's articles relevant to Star Forest, but I am still unclear
>>>>>>>> about the exact relation & graphical notation of roots & leaves in SF if
>>>>>>>> it's the case of DMPlex solution arrays.
>>>>>>>>
>>>>>>>> No, the number of roots for a DMPlex is the number of mesh points
>>>>>>>> in the local portion of the mesh
>>>>>>>>
>>>>>>>> > (2) If it is so, there is an issue that I cannot define "root
>>>>>>>> data" and "leave data" generally. I am trying to following
>>>>>>>> "src/vec/is/sf/tutorials/ex1f.F90", however, in that example, size of roots
>>>>>>>> and leaves are predefined as 6. How can I generalize that? Because I can
>>>>>>>> get size of leaves using DAG depth(or height), which is equal to number of
>>>>>>>> vertices each proc has. But, how can I get the size of my "roots" region
>>>>>>>> from SF? Any example about that? This question is connected to how can I
>>>>>>>> define "rootdata" for "PetscSFBcastBegin/End()".
>>>>>>>>
>>>>>>>> Does the diagram help you generalize?
>>>>>>>>
>>>>>>>> > (3) More importantly, with the attached PetscSection & SF layout,
>>>>>>>> my vector is only resolved for the size equal to "number of roots" for each
>>>>>>>> proc, but not for the overlapping area(i.e., "leaves"). What I wish to do
>>>>>>>> is to exchange (or reduce) the solution data between each proc, in the
>>>>>>>> overlapping region. Can I get some advices why my vector does not encompass
>>>>>>>> the "leaves" regime? Is there any example doing similar things?
>>>>>>>> Going back to my first response: if you use a section to say how
>>>>>>>> many pieces of data are associated with each local mesh point, then a
>>>>>>>> PetscSF is constructed that requires no more manipulation from you.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, May 22, 2022 at 10:47 AM Mike Michell <
>>>>>>>> mi.mike1021 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thank you for the reply.
>>>>>>>>> The PetscSection and PetscSF objects are defined as in the
>>>>>>>>> attached mock-up code (Q_PetscSF_1.tar). 1-DOF is defined on vertex as my
>>>>>>>>> solution is determined on each vertex with 1-DOF from a finite-volume
>>>>>>>>> method.
>>>>>>>>>
>>>>>>>>> As follow up questions:
>>>>>>>>> (1) It seems that the "roots" means the number of vertex not
>>>>>>>>> considering overlap layer, and "leaves" seems the number of distributed
>>>>>>>>> vertex for each processor that includes overlap layer. Can you acknowledge
>>>>>>>>> that this is correct understanding? I have tried to find clearer examples
>>>>>>>>> from PETSc team's articles relevant to Star Forest, but I am still unclear
>>>>>>>>> about the exact relation & graphical notation of roots & leaves in SF if
>>>>>>>>> it's the case of DMPlex solution arrays.
>>>>>>>>>
>>>>>>>>> (2) If it is so, there is an issue that I cannot define "root
>>>>>>>>> data" and "leave data" generally. I am trying to following
>>>>>>>>> "src/vec/is/sf/tutorials/ex1f.F90", however, in that example, size of roots
>>>>>>>>> and leaves are predefined as 6. How can I generalize that? Because I can
>>>>>>>>> get size of leaves using DAG depth(or height), which is equal to number of
>>>>>>>>> vertices each proc has. But, how can I get the size of my "roots" region
>>>>>>>>> from SF? Any example about that? This question is connected to how can I
>>>>>>>>> define "rootdata" for "PetscSFBcastBegin/End()".
>>>>>>>>>
>>>>>>>>> (3) More importantly, with the attached PetscSection & SF layout,
>>>>>>>>> my vector is only resolved for the size equal to "number of roots" for each
>>>>>>>>> proc, but not for the overlapping area(i.e., "leaves"). What I wish to do
>>>>>>>>> is to exchange (or reduce) the solution data between each proc, in the
>>>>>>>>> overlapping region. Can I get some advices why my vector does not encompass
>>>>>>>>> the "leaves" regime? Is there any example doing similar things?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Fri, May 20, 2022 at 4:45 PM Mike Michell <
>>>>>>>>>> mi.mike1021 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>
>>>>>>>>>>> > "What I want to do is to exchange data (probably just
>>>>>>>>>>> MPI_Reduce)" which confuses me, because halo exchange is a point-to-point
>>>>>>>>>>> exchange and not a reduction.  Can you clarify?
>>>>>>>>>>> PetscSFReduceBegin/End seems to be the function that do
>>>>>>>>>>> reduction for PetscSF object. What I intended to mention was either
>>>>>>>>>>> reduction or exchange, not specifically intended "reduction".
>>>>>>>>>>>
>>>>>>>>>>> As a follow-up question:
>>>>>>>>>>> Assuming that the code has its own local solution arrays (not
>>>>>>>>>>> Petsc type), and if the plex's DAG indices belong to the halo region are
>>>>>>>>>>> the only information that I want to know (not the detailed section
>>>>>>>>>>> description, such as degree of freedom on vertex, cells, etc.). I have
>>>>>>>>>>> another PetscSection for printing out my solution.
>>>>>>>>>>> Also if I can convert that DAG indices into my local cell/vertex
>>>>>>>>>>> index, can I just use the PetscSF object created from DMGetPointSF(),
>>>>>>>>>>> instead of "creating PetscSection + DMGetSectionSF()"? In other words, can
>>>>>>>>>>> I use the PetscSF object declared from DMGetPointSF() for the halo
>>>>>>>>>>> communication?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> No, because that point SF will index information by point number.
>>>>>>>>>> You would need to build a new SF that indexes your dofs. The steps you would
>>>>>>>>>> go through are exactly the same as you would if you just told us
>>>>>>>>>> what the Section is that indexes your data.
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>      Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Mike
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The PetscSF that is created automatically is the "point sf" (
>>>>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMGetPointSF/): it
>>>>>>>>>>>> says which mesh points (cells, faces, edges and vertices) are duplicates of
>>>>>>>>>>>> others.
>>>>>>>>>>>>
>>>>>>>>>>>> In a finite volume application we typically want to assign
>>>>>>>>>>>> degrees of freedom just to cells: some applications may only have one
>>>>>>>>>>>> degree of freedom, others may have multiple.
>>>>>>>>>>>>
>>>>>>>>>>>> You encode where you want degrees of freedom in a PetscSection
>>>>>>>>>>>> and set that as the section for the DM in DMSetLocalSection() (
>>>>>>>>>>>> https://petsc.org/release/docs/manualpages/DM/DMSetLocalSection.html
>>>>>>>>>>>> )
>>>>>>>>>>>>
>>>>>>>>>>>> (A c example of these steps that sets degrees of freedom for
>>>>>>>>>>>> *vertices* instead of cells is `src/dm/impls/plex/tutorials/ex7.c`)
>>>>>>>>>>>>
>>>>>>>>>>>> After that you can call DMGetSectionSF() (
>>>>>>>>>>>> https://petsc.org/main/docs/manualpages/DM/DMGetSectionSF/) to
>>>>>>>>>>>> the the PetscSF that you want for halo exchange: the one for your solution
>>>>>>>>>>>> variables.
>>>>>>>>>>>>
>>>>>>>>>>>> After that, the only calls you typically need in a finite
>>>>>>>>>>>> volume code is PetscSFBcastBegin() to start a halo exchange and
>>>>>>>>>>>> PetscSFBcastEnd() to complete it.
>>>>>>>>>>>>
>>>>>>>>>>>> You say
>>>>>>>>>>>>
>>>>>>>>>>>> > What I want to do is to exchange data (probably just
>>>>>>>>>>>> MPI_Reduce)
>>>>>>>>>>>>
>>>>>>>>>>>> which confuses me, because halo exchange is a point-to-point
>>>>>>>>>>>> exchange and not a reduction.  Can you clarify?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, May 20, 2022 at 8:35 PM Mike Michell <
>>>>>>>>>>>> mi.mike1021 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear PETSc developer team,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, I am using DMPlex for a finite-volume code and trying to
>>>>>>>>>>>>> understand the usage of PetscSF. What is a typical procedure for doing halo
>>>>>>>>>>>>> data exchange at parallel boundary using PetscSF object on DMPlex? Is there
>>>>>>>>>>>>> any example that I can refer to usage of PetscSF with distributed DMPlex?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Assuming to use the attached mock-up code and mesh, if I give
>>>>>>>>>>>>> "-dm_distribute_overlap 1 -over_dm_view" to run the code, I can see a
>>>>>>>>>>>>> PetscSF object is already created, although I have not called
>>>>>>>>>>>>> "PetscSFCreate" in the code. How can I import & use that PetscSF already
>>>>>>>>>>>>> created by the code to do the halo data exchange?
>>>>>>>>>>>>>
>>>>>>>>>>>>> What I want to do is to exchange data (probably just
>>>>>>>>>>>>> MPI_Reduce) in a parallel boundary region using PetscSF and its functions.
>>>>>>>>>>>>> I might need to have an overlapping layer or not.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Mike
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220603/4f3ec0e5/attachment-0001.html>

More information about the petsc-users mailing list