[petsc-users] DMPlex: a Mapping Array between Natural and Distributed Cell Index

Matthew Knepley knepley at gmail.com
Fri Jul 15 09:09:24 CDT 2022 

On Fri, Jul 15, 2022 at 8:46 AM Bora Jeong <boraj1021 at gmail.com> wrote:

> Okay, I got it what's happening. First, this very bright functionality of
> petsc (the natural-global converting) needs to be documented in a better
> way for sure. Currently, it is very difficult to use/follow this features
> as an external user. Hope this will move forward in a better way.
>
> Then next question is if I load/distribute mesh just like I am doing right
> now shown in the sample code, can I assume that my code does not create
> natural sf during the distribution(always)? In other words, can I
> always get the natural order of each node by simply stacking the preceding
> processor's number of node? For example, for proc=0, natural node ID might
> be just 1 to nnode_proc_0,
> for proc=1, it might be {nnode_proc_0 + 1 to nnode_proc_0 + nnode_proc_1}
> and so on.
>
> Does that always make sense in this case?
>

No, but if you call DMPlexDistribute() yourself, rather than having it
called automatically by DMSetFromOptions(), you can
preserve the mapping:

  https://petsc.org/main/docs/manualpages/DMPLEX/DMPlexDistribute/

The SF returned maps the original point distribution, which is in the same
order as the file, to the redistributed points, which are determined
by the mesh partitioner.

  Thanks,

      Matt


> Best
>
>
> On Fri, Jul 15, 2022 at 8:07 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Jul 15, 2022 at 7:17 AM Bora Jeong <boraj1021 at gmail.com> wrote:
>>
>>> A sample code for loading dm and declaring Natural SF from a sphere mesh
>>> is attached here. PetscSFView() returns NULL from sf_nat.
>>>
>>
>> The Global-to-Natural mapping relates global dofs to natural dofs. Thus,
>> in order to compute this mapping the DM has to know
>> about the dof layout before distribution. This means you need to setup a
>> local section before distributing, as we do for example in
>> Plex test ex15. This makes things more complicated since everything
>> cannot be packaged up into DMSetFromOptions(), but we need
>> user input so I do not see a simpler way to do things.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Best
>>>
>>> On Fri, Jul 15, 2022 at 6:39 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Jul 14, 2022 at 8:25 PM Bora Jeong <boraj1021 at gmail.com> wrote:
>>>>
>>>>> Okay, I checked it and you are correct. In my case, simply, natural
>>>>> node index can be identified by stacking all the preceding processor's
>>>>> numbers of nodes for a particular processor, which is good due to
>>>>> simplicity. However, one serious question is why this is happening in my
>>>>> code? In other words, why the natural SF is not created during the mesh
>>>>> distribution? My code wants to have consistency in dealing with this
>>>>> natural indexing for several different kinds of mesh files. So there is a
>>>>> necessity to guarantee of consistency in this weird behavior.
>>>>>
>>>>
>>>> I can't tell what is going on in your code unless I can run it. Do you
>>>> have a simple example?
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Best,
>>>>>
>>>>> On Thu, Jul 14, 2022 at 6:43 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Thu, Jul 14, 2022 at 5:47 PM Bora Jeong <boraj1021 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you for the comments. I have these errors when I call
>>>>>>> PetscSFView() after DMGetNaturalSF()
>>>>>>>
>>>>>>> [2]PETSC ERROR: --------------------- Error Message
>>>>>>> --------------------------------------------------------------
>>>>>>> [2]PETSC ERROR: Null argument, when expecting valid pointer
>>>>>>> [2]PETSC ERROR: Null Pointer: Parameter # 1
>>>>>>> [2]PETSC ERROR: See https://petsc.org/release/faq/ for trouble
>>>>>>> shooting.
>>>>>>> [2]PETSC ERROR: Petsc Release Version 3.17.0, unknown
>>>>>>> [2]PETSC ERROR: #1 PetscSFView() at
>>>>>>> [2]PETSC ERROR: #2 User provided function() at User file:0
>>>>>>> Abort(85) on node 2 (rank 0 in comm 16): application called
>>>>>>> MPI_Abort(MPI_COMM_SELF, 85) - process 0
>>>>>>>
>>>>>>
>>>>>> Clearly NULL was returned, which means no natural SF was created.
>>>>>>
>>>>>>
>>>>>>> Below is the structure to load a mesh file from gmsh;
>>>>>>>
>>>>>>>   call DMCreate(PETSC_COMM_WORLD, dm, ierr);CHKERRA(ierr)
>>>>>>>   call DMSetType(dm, plex, ierr);CHKERRA(ierr)
>>>>>>>   call DMSetUseNatural(dm, PETSC_TRUE, ierr);CHKERRA(ierr)
>>>>>>>   call DMSetFromOptions(dm, ierr);CHKERRA(ierr)
>>>>>>>   call DMGetNaturalSF(dm, sf_nat, ierr);CHKERRA(ierr)
>>>>>>>   call PetscSFView(sf_nat, PETSC_VIEWER_STDOUT_WORLD,
>>>>>>> ierr);CHKERRA(ierr)
>>>>>>>
>>>>>>
>>>>>> The natural SF is created during mesh distribution. That has not
>>>>>> happened here. This means that
>>>>>> the order of cells is identical to the file it was read from.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>       Matt
>>>>>>
>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> On Thu, Jul 14, 2022 at 10:49 AM Matthew Knepley <knepley at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Wed, Jul 13, 2022 at 10:17 PM Bora Jeong <boraj1021 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear petsc team,
>>>>>>>>>
>>>>>>>>> I am a user of DMPlex for a finite volume code and there is a
>>>>>>>>> necessity to know global index of each cell. Here the global index means
>>>>>>>>> the indexing that can be found from a mesh file itself without distribution
>>>>>>>>> over processors. It seems petsc community denotes this indexing term as
>>>>>>>>> "natural".
>>>>>>>>>
>>>>>>>>> What I want to do is to create a local array (not petsc vector but
>>>>>>>>> just an array variable in the program) to map distributed cell ID to natual
>>>>>>>>> cell ID, for example, an array "A";
>>>>>>>>> A(distributed_node_ID) = natural_node_ID
>>>>>>>>>
>>>>>>>>> There are some petsc functions to support mapping between global
>>>>>>>>> and natural vectors. However, I just need to define the array "A" as above
>>>>>>>>> example. To achieve this, what is a proper/smart way? In other words, how
>>>>>>>>> can I extract the natural_cell_ID from a distributed local_cell_ID?
>>>>>>>>>
>>>>>>>>> I turned on DMSetUseNatural(DM, PETSC_TRUE) before distribution,
>>>>>>>>> but after this, defining all the required section and star forest objects
>>>>>>>>> to get natural and global vectors seems not that direct way for my purpose,
>>>>>>>>> which is just to extract the above mapping array "A". Can I get any
>>>>>>>>> comments about it?
>>>>>>>>>
>>>>>>>>
>>>>>>>> There is only one thing created, the sfNatural PetscSF object,
>>>>>>>> which you can get with DMGetNaturalSF(). The roots of this SF are
>>>>>>>> the global numbers of dofs stored in PETSc vectors, and the leaves
>>>>>>>> are natural numbers for these dofs. Thus, when we map global
>>>>>>>> vectors to natural vectors in DMPlexGlobalToNaturalBegin/End(), we
>>>>>>>> call PetscSFBcastBegin/End(). Mapping natural to global we call
>>>>>>>> PetscSFReduceBegin/End(). You could pull the information out of the
>>>>>>>> SF using PetscSFGetGraph() if you want.
>>>>>>>>
>>>>>>>>   Thanks,
>>>>>>>>
>>>>>>>>     Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Mo
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220715/e8474545/attachment.html>

More information about the petsc-users mailing list