[petsc-users] Vector including ghost pattern

[Denner, Fabian]

Hi everyone, 

thanks Barry you are right, I messed it up. The setVecGhost() function is my own function. Find below a revised explanation of my problem and an adjusted and simplified example. 

Some remarks for a clearer understanding. The connectivity information of the mesh is generally stored in a pair of arrays, ia[] and ja[] (similar to example "dm/ao/examples/tutorials/ex2.c"). The array ja[] contains the indices of the neighbor elements. The array ia[] contains the start-position in array ja[] for each element and has the length numberOfElements+1. For instance, if there are 3 elements the connectivity information could look as followed: ja[] is [45 12 34 8 101 23 4 49 95] and ia[] is [0 2 5 9]. This means the elements with the indices 45 and 12 are neighbors of the first element, the elements 34, 8 and 101 are neighbors of the second element and so on. To be able to loop over the neighbor elements the array ia[] needs to contain numberOfElements+1 positions, with the total number of neighbors stored in the last position. 
Since the mesh handling is supposed to work in parallel the mesh is distributed to several processors. Therefore, the mesh is stored in "local" arrays, containing the mesh (vertices, faces, elements) information which is local to that processor, and in "ghost" arrays, containing the mesh information of neighboring entities stored on a different processor. To use for example the VecScatter functions I would like to cast this connectivity information into vectors. Unfortunately I couldn't find an opportunity to set the "ghost" information in the vector for array ia[] right, since it gathers the information from the "global" representation and thus I can't loop over the ghost elements anymore because the neighbors of a particular element are most likely stored in a different position of the "global" array than in the "ghost" array.

VecCreateGhost(PETSC_COMM_WORLD, (1+numberLocalElements), PETSC_DECIDE, (1 + numberGhostElements), iaGhost, &vec);

VecGetOwnershipRange(vec, &start, &end);
for (i = start; i < end; i++)
{
    value = (double) iaGlobal[i];
    VecSetValues(vec, 1, &i, &value, INSERT_VALUES);
}

VecAssemblyBegin(vec);
VecAssemblyEnd(vec);

VecGhostUpdateBegin(vec, INSERT_VALUES, SCATTER_FORWARD);
VecGhostUpdateEnd(vec, INSERT_VALUES, SCATTER_FORWARD);


Does anyone have an idea how to solve the problem or did anyone have a similar problem before?

Best regards and thanks in advance,
Fabian

________________________________________
From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith [bsmith at mcs.anl.gov]
Sent: 17 July 2010 03:48
To: PETSc users list
Subject: Re: [petsc-users] Vector including ghost pattern

On Jul 16, 2010, at 11:27 AM, Denner, Fabian wrote:

> Hi.
>
> Let me try to explain it with a more detailed example.
> I have the two arrays of my local element connectivity information (local elements are elements stored on the current rank), with the neighbor element indices in array jaLocal[] and the "start"-position for each element in jaLocal[] stored in array iaLocal.
>
>
> jaLocal: [3 6 17 23 38 9 0 5 6 8 10 11 ...]
> iaLocal: [0 3 7 11 13 17 21 ....]
>
> The same I have for my ghost elements. I added also a sample of the ghost element index array to explain it a bit better.
>
> indexGhost: [256 453 301 ...]
>
> jaGhost: [465 204 461 299 213 ....]
> iaGhost: [0 4 8 12 16 19 ....]
>
> If I like to cast this data in to a vector with ghost pattern, I have to call the following commands (which could look like that):
>
> VecCreateGhost(PETSC_COMM_WORLD, (1 + elemLocal.n), PETSC_DECIDE, (1 + elemGhost.n), tempPtr, &elemVec.neighborPtr);

   What is tempPtr?

> setVecGhost(elemGlobal.neighborPtr, elemVec.neighborPtr, size, rank);

   I have no idea what the function setVecGhost() is? Is it in PETSc, where? Something else.

   Barry

>
> To set the ghost values, I have to call setVecGhost with a global (all elements, not only local) assembly of my data, which gathers the right data.
> Let's go back to the sample arrays of the ghost elements. Obviously, the positions of the neighbor elements of element 256 (which is the first element in the ghost arrays) does not start at position 0 in jaGlobal[] (an equivalent global array of the connectivity) but for instance at position 1087. Therefore, the vector creation fails or the a segmentation violation appears.
>
> Hopefully, this description was a bit clearer.
>
> Thanks a lot!
> Fabian
>
> ________________________________________
> From: petsc-users-bounces at mcs.anl.gov [petsc-users-bounces at mcs.anl.gov] On Behalf Of Barry Smith [bsmith at mcs.anl.gov]
> Sent: 16 July 2010 16:56
> To: PETSc users list
> Subject: Re: [petsc-users] Vector including ghost pattern
>
> On Jul 16, 2010, at 10:31 AM, Denner, Fabian wrote:
>
>> Hi everybody,
>>
>> I have the following problem with the handling of a 3D mesh with mixed element types. I aim to build vectors containing the ghost pattern for the mesh of a parallel CFD simulation. Inspired by the example "dm/ao/examples/tutorials/ex2.c" the connectivity information, represented by the element neighbors, is stored in two arrays - one array ja[] containing the neighbor element indices and the other array ia[] (of length n+1) containing the "start" index for each element in array ja[] (compare example below or adjacency graph in "dm/ao/examples/tutorials/ex2.c"). The neighbors of my local elements, elements assigned to the present process, and the neighbors of my ghost elements are stored in different pairs of arrays.
>>
>> Element indices: index[0 1 2 3 4 5 6 7 .... n]
>> Neighbour element indices: ja[3 5 6 11 57 311 41 0 5 67 ..... 532 106]
>> "Start"-position of neighour indices in array ja[]: ia[0 3 7 11 13 .... 1503]
>>
>> The issue appears when I try to cast the arrays in a vector with ghost pattern. When I create a vector with VecCreateGhost, deliver my global neighbor arrays and update my ghost pattern, the data will be lost, since the number of neighbors varies for each element.
>
>   Hmm, VecCreateGhost() doesn't say anything about the number of neighbors or elements. You are free to set any number of ghost points associated with things. So it is up to you to assign the appropriate ghost points etc.
>
>   Perhaps if you are a bit more explicit in your question we could answer it better?
>
>   Barry
>
>> Subsequently, I can't loop over my ghost neighbors anymore. Is there any different handling of connectivity/adjacency information or is there an opportunity to "prepare" the data in an alternative way?
>>
>> Best regards,
>> Fabian Denner
>