[petsc-users] Consistent domain decomposition between DMDA and DMPLEX

Thu Mar 28 21:08:21 CDT 2019

On Thu, Mar 28, 2019 at 8:51 PM Swarnava Ghosh <swarnava89 at gmail.com> wrote:

> "Is this a finite element Poisson solve on your coarse unstructured mesh?"
> > Yes, this is a finite element Poisson solve.
>

You know the vertex division from DMDA. You can easily make a cell
partition by assigning
a cell to the process of its first vertex (or any vertex really). You can
do this by using a
PlexPartitionerShell as in the example I linked. However, you also want to
force
the vertex partition afterwards,  so you can copy values locally.

In DMPlexCreatePointSF(), we decide who will own the parts of the boundary.
Normally, everyone
votes for the points that they have, and the highest rank process wins (we
also have a randomized
version). However, here you can just substitute a function that only votes
for a vertex if that vertex
is in its DMDA partition. Would not be more than an hour of coding/testing
I think. Could you guys
make a small example?

  Thanks,

     Matt

Sincerely,
> Swarnava
>
> On Thu, Mar 28, 2019 at 5:09 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Mar 28, 2019 at 7:24 PM Swarnava Ghosh <swarnava89 at gmail.com>
>> wrote:
>>
>>> Hi Mark and Matt,
>>>
>>> I calculate my unknown fields at the nodes of my coarse unstructured
>>> mesh, and then project the solution at some of the fine structured mesh
>>> nodes.
>>> The only global matrix I form is on the unstructured coarse mesh to do a
>>> Poisson solve.
>>>
>>
>> Is this a finite element Poisson solve on your coarse unstructured mesh?
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Sincerely,
>>> Swarnava
>>>
>>> On Thu, Mar 28, 2019 at 5:56 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>>
>>>>>
>>>>> That seems like a bad tradeoff. You avoid one communication during
>>>>> injection for at least that much or more during
>>>>> FE assembly on that cell partition?
>>>>>
>>>>>
>>>> I am just guessing about the purpose as a way to describing what they
>>>> are asking for.
>>>>
>>>>
>>>>>   Matt
>>>>>
>>>>>
>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>
>>>>>>>> Sincerely,
>>>>>>>> Swarnava
>>>>>>>>
>>>>>>>> On Fri, Mar 22, 2019 at 4:08 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>>
>>>>>>>>> Matt,
>>>>>>>>> I think they want a vertex partitioning. They may have elements on
>>>>>>>>> the unstructured mesh that intersect with any number of processor domains
>>>>>>>>> on the structured mesh. But the unstructured mesh vertices are in the
>>>>>>>>> structured mesh set of vertices. They want the partition of the
>>>>>>>>> unstructured mesh vertices (ie, matrices) to be slaved to the partitioning
>>>>>>>>> of the structured mesh.
>>>>>>>>> Do I have that right Swarnava?
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>> On Fri, Mar 22, 2019 at 6:56 PM Matthew Knepley via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> On Thu, Mar 21, 2019 at 8:20 PM Swarnava Ghosh via petsc-users <
>>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear PETSc users and developers,
>>>>>>>>>>>
>>>>>>>>>>> I am new to DMPLEX and had a query regarding setting up a
>>>>>>>>>>> consistent domain decomposition of two meshes in PETSc.
>>>>>>>>>>> I have a structured finite difference grid, managed through
>>>>>>>>>>> DMDA. I have another unstructured finite element mesh managed through
>>>>>>>>>>> DMPLEX. Now all the nodes in the unstructured finite element mesh also
>>>>>>>>>>> belong to the set of nodes in the structured finite difference mesh (but
>>>>>>>>>>> not necessarily vice-versa), and the number of nodes in DMPLEX mesh is less
>>>>>>>>>>> than the number of nodes in DMDA mesh. How can I guarantee a consistent
>>>>>>>>>>> domain decomposition of the two meshes? By consistent, I mean that if a
>>>>>>>>>>> process has a set of nodes P from DMDA, and the same process has the set of
>>>>>>>>>>> nodes Q from DMPLEX, then Q is a subset of P.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Okay, this is not hard. DMPlexDistribute() basically distributes
>>>>>>>>>> according to a cell partition. You can use PetscPartitionerShell() to stick
>>>>>>>>>> in whatever cell partition you want. You can see me doing this here:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://bitbucket.org/petsc/petsc/src/e2aefa968a094f48dc384fffc7d599a60aeeb591/src/dm/impls/plex/examples/tests/ex1.c#lines-261
>>>>>>>>>>
>>>>>>>>>> Will that work for you?
>>>>>>>>>>
>>>>>>>>>>   Thanks,
>>>>>>>>>>
>>>>>>>>>>     Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I look forward to your response.
>>>>>>>>>>>
>>>>>>>>>>> Sincerely,
>>>>>>>>>>> Swarnava
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190328/10a4139b/attachment.html>