[petsc-users] Memory Distribution and Mesh Partition in Finite Element Method

[Matthew Knepley]

On Tue, Apr 26, 2016 at 11:00 PM, Jie Cheng <chengj5 at rpi.edu> wrote:

> Hello Barry
>
> It turns out METIS can easily generate the necessary graph information
> from the type of connectivity file I described with mesh2dual and
> mesh2nodal commands. But I do not fully understand the trick you talked
> about:
>
> Suppose I have 10 elements and I am running my program with 2 MPI ranks.
>
> 1) I should let rank 0 take care of the first to the 5th element, and rank
> 1 take care of the 6th to the 10th element and let PETSc do the real
> partitioning?
>

No, Barry says

  "first partition the element across processes, then partition the
vertices (nodal values) subservient to the partitioning of the elements"

meaning only give a vertex to a process if it already owns the element
containing this vertex.


> 2) I have the necessary graph information, but how do I partition the
> elements and the vertices? Call MatCreateMPIAdj twice to create two isgs? I
> do not understand what you meant by “partition the vertices (nodal values)
> subservient to the partitioning of the elements”.
>

If you used PETSc to do it, you would create an MPIAdj which the element
adjacency information (which I assume you got from mesh2dual).
We would partition the elements by calling parmetis underneath. Then it
would be up to you to partition vertices, just as it is when calling
parmetis by hand.


> 3) To set values in the global stiffness matrix I need to know the global
> element number and global vertex number, how to get them?
>

You, of course, start with these numbers. As Barry says, in order to make
the dofs contiguous you should renumber the cells and vertices
so that each process owns a contiguous block.

  Thanks,

     Matt


> I have to ask these in detail because the manual seems to be too brief
> about the approach. Thanks for your patience.
>
> Best
> Jie
>
>
> > On Apr 26, 2016, at 7:56 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> >> On Apr 26, 2016, at 6:50 PM, Jie Cheng <chengj5 at rpi.edu> wrote:
> >>
> >> Hi Barry
> >>
> >> Thanks for your answer. But given the mesh file I have, how do I
> partition the elements and vertices? MatCreateMPIAdj requires the graph
> information on adjacent elements (to set up ia and ja). Do I have to use
> more sophisticated meshing software to create mesh files that have graph
> information in it?
> >
> >   If you don't have any neighbor information then you cannot do any
> partitioning. So you need a way to get or create the neighbor information.
> >
> > Barry
> >
> >>
> >> Jie
> >>
> >>> On Apr 26, 2016, at 2:18 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>>
> >>>
> >>> The "trick" is that first partition the element across processes, then
> partition the vertices (nodal values) subservient to the partitioning of
> the elements and then you "renumber" the elements and vertices so that the
> elements on the first process are numbered first, followed by the elements
> on the second process etc and similarly the vertices on the first process
> are numbered before the vertices on the second processes etc.
> >>>
> >>> Now each process just needs to loop over its elements compute the
> element stiffness/load and call MatSetValues/VecSetValues() the the "new"
> numbering of the vertices.  The "old" numbering that was on the disk is
> simply not used in communicating with PETSc, you only use the new PETSc
> numbering.
> >>>
> >>> Barry
> >>>
> >>>> On Apr 26, 2016, at 1:03 PM, Jie Cheng <chengj5 at rpi.edu> wrote:
> >>>>
> >>>> Hello everyone
> >>>>
> >>>> I have a finite element code to solve nonlinear solid mechanics
> (using Newton-Raphson iteration) implemented with PETSc. The code is
> serial, organized as following:
> >>>>
> >>>> 1) Read the connectivity of the unstructured mesh, coordinates of
> nodes from individual txt files.
> >>>> 1.1) Connectivity file contains [# of elements] rows, and each row
> lists the global number of nodes on that element. Take 3d hexahedral
> elements for instance:
> >>>>    223 224 298 297 1 2 76 75
> >>>>    224 225 299 298 2 3 77 76
> >>>>    … …
> >>>> 1.2) Coordinates file contains [# of nodes] rows, and each row lists
> the coordinates of a node, for example:
> >>>>    0                   0.0011                      3.9e-5
> >>>>    2.3677e-5     0.001.9975                3.9e-5
> >>>>    … …
> >>>>
> >>>> 2) Create the global stiffness A matrix with MatCreateSeqAIJ since
> the dimensions and nonzero pattern are known from the connectivity.
> >>>> 3) Loop over the element to compute the element stiffness matrix and
> right hand side. Then assemble the global stiffness matrix and right hand
> side.
> >>>> 4) Solve the linear equation with KSPsolve for the displacement
> increment, then go back to Step 3.
> >>>>
> >>>> The code works fine in serial, now I’m trying to parallelize it. To
> partition the mesh, I can use partdmesh from METIS, or let PETSc calls it.
> Either way I will find a way to assign different elements and nodes to
> different ranks. My question is: since PETSc does not allow us to control
> the memory distribution of the parallel matrix/vector, how do I make sure
> the rank happens to have all/most memory it needs for the specific
> elements? For example, rank 0 is in charged of element n, and needs to
> insert values to A[ i ][ j ], how do I make sure the i-th row is assigned
> to rank 0?
> >>>>
> >>>> This question is fundamental for people work with finite element
> methods. I checked the tutorial codes but did not find an example to
> follow. Section 3.5 of the manual talks about partition, but it does not
> say anything about linking the partition with the memory distribution.
> Could anyone give me some instructions on this issue? Thank you in advance!
> >>>>
> >>>> Best
> >>>> Jie Cheng
> >>>
> >>
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160427/192ee084/attachment-0001.html>