[petsc-users] Vector including ghost pattern

[Denner, Fabian]

Hi.

Let me try to explain it with a more detailed example. 
I have the two arrays of my local element connectivity information (local elements are elements stored on the current rank), with the neighbor element indices in array jaLocal[] and the "start"-position for each element in jaLocal[] stored in array iaLocal.


jaLocal: [3 6 17 23 38 9 0 5 6 8 10 11 ...]
iaLocal: [0 3 7 11 13 17 21 ....]

The same I have for my ghost elements. I added also a sample of the ghost element index array to explain it a bit better.

indexGhost: [256 453 301 ...]

jaGhost: [465 204 461 299 213 ....]
iaGhost: [0 4 8 12 16 19 ....]

If I like to cast this data in to a vector with ghost pattern, I have to call the following commands (which could look like that):

VecCreateGhost(PETSC_COMM_WORLD, (1 + elemLocal.n), PETSC_DECIDE, (1 + elemGhost.n), tempPtr, &elemVec.neighborPtr);
setVecGhost(elemGlobal.neighborPtr, elemVec.neighborPtr, size, rank);

To set the ghost values, I have to call setVecGhost with a global (all elements, not only local) assembly of my data, which gathers the right data. 
Let's go back to the sample arrays of the ghost elements. Obviously, the positions of the neighbor elements of element 256 (which is the first element in the ghost arrays) does not start at position 0 in jaGlobal[] (an equivalent global array of the connectivity) but for instance at position 1087. Therefore, the vector creation fails or the a segmentation violation appears.

Hopefully, this description was a bit clearer.

Thanks a lot!
Fabian

________________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith [bsmith at mcs.anl.gov]
Sent: 16 July 2010 16:56
To: PETSc users list
Subject: Re: [petsc-users] Vector including ghost pattern

On Jul 16, 2010, at 10:31 AM, Denner, Fabian wrote:

> Hi everybody,
>
> I have the following problem with the handling of a 3D mesh with mixed element types. I aim to build vectors containing the ghost pattern for the mesh of a parallel CFD simulation. Inspired by the example "dm/ao/examples/tutorials/ex2.c" the connectivity information, represented by the element neighbors, is stored in two arrays - one array ja[] containing the neighbor element indices and the other array ia[] (of length n+1) containing the "start" index for each element in array ja[] (compare example below or adjacency graph in "dm/ao/examples/tutorials/ex2.c"). The neighbors of my local elements, elements assigned to the present process, and the neighbors of my ghost elements are stored in different pairs of arrays.
>
> Element indices: index[0 1 2 3 4 5 6 7 .... n]
> Neighbour element indices: ja[3 5 6 11 57 311 41 0 5 67 ..... 532 106]
> "Start"-position of neighour indices in array ja[]: ia[0 3 7 11 13 .... 1503]
>
> The issue appears when I try to cast the arrays in a vector with ghost pattern. When I create a vector with VecCreateGhost, deliver my global neighbor arrays and update my ghost pattern, the data will be lost, since the number of neighbors varies for each element.

   Hmm, VecCreateGhost() doesn't say anything about the number of neighbors or elements. You are free to set any number of ghost points associated with things. So it is up to you to assign the appropriate ghost points etc.

   Perhaps if you are a bit more explicit in your question we could answer it better?

   Barry

> Subsequently, I can't loop over my ghost neighbors anymore. Is there any different handling of connectivity/adjacency information or is there an opportunity to "prepare" the data in an alternative way?
>
> Best regards,
> Fabian Denner