[petsc-users] Accessing Vector's ghost values

[Barry Smith]

I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone:

    To update the ghost values from the locations on the other processes one must call 
    VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal
    usage is 
$     VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD);
$     VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD);
$     VecGhostGetLocalForm(x,&xlocal);
$     VecGetArray(xlocal,&xvalues);
$        /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */
$     VecRestoreArray(xlocal,&xvalues);
$     VecGhostRestoreLocalForm(x,&xlocal);


On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:

> Yeeee-ha!  
> 
> VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.
> 
> Thank you Mohammad, thank you all who helped me today!
> 
> 
>     Cheers
> 
> 
>     Bojan
> 
> 
> 
> On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:
>> You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56
>> 
>> Mohammad
>> 
>> On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <bojan.niceno at psi.ch> wrote:
>> On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:
>>> based on,
>>> 
>>> VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);
>>> 
>>> 
>>> it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?
>> 
>> I tried that before.  I did:
>> 
>> Vec lx;
>> VecGhostGetLocalForm(x, &lx)
>> 
>> then I copied "lx" to my variable, like
>> 
>> for(Int i=1; i<N; i++)    /* N includes buffer cells */
>>   unk[i] = lx[i]
>> 
>> but ghost values were also zero.   I am thinking that PETSc somehow clears the ghost values after a call to KSP.  Is it the case?
>> 
>> 
>>     Kind regards,
>> 
>> 
>>     Bojan
>> 
>> 
>>> 
>>> On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <bojan.niceno at psi.ch> wrote:
>>> Dear Mohammad,
>>> 
>>> 
>>> it doesn't help me, or I did not understand your explanation.
>>> 
>>> If I do this:
>>> 
>>>   /* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */
>>> 
>>>   for(Int i=0; i<n; i++) {
>>>     Int gi = mesh.nodes[i].global_number;
>>>     VecGetValues(x, 1, &gi, &unk[i]);
>>>   }
>>> 
>>>   /* copy ghost values (CREATES MANY WARNINGS */
>>> 
>>>   for(Int i=n; i<N; i++) {
>>>     VecGetValues(x, 1, &i, &unk[i]);
>>>   }
>>> 
>>> I get arnings are like this.
>>> 
>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Argument out of range!
>>> [0]PETSC ERROR: Can only get local values, trying 3518!
>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> [3]PETSC ERROR: Argument out of range!
>>> [3]PETSC ERROR: Can only get local values, trying 3511!
>>> [3]PETSC ERROR: ------------------------------------------------------------------------
>>> 
>>> What am I doing wrong here?
>>> 
>>> 
>>>     Cheers,
>>> 
>>> 
>>>     Bojan
>>> 
>>> 
>>> 
>>> On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:
>>>> just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'
>>>> 
>>>> On Feb 23, 2012 1:16 PM, "Bojan Niceno" <bojan.niceno at psi.ch> wrote:
>>>> Dear Jed,
>>>> 
>>>> thanks.
>>>> 
>>>> Now I have the following:
>>>> 
>>>> - Array unk, which should hold values inside the partition and in ghost cells.  It is big enough to hold both
>>>> - Vec x, created by command VecCreateGhost, with proper padding for ghost cells
>>>> - Successful call to linear solver in parallel.
>>>> 
>>>> But I need to copy ghost values from x to my array unk. 
>>>> 
>>>> How can I do it?
>>>> 
>>>> 
>>>>     Kind regards,
>>>> 
>>>> 
>>>>     Bojan
>>>> 
>>>> On 2/23/2012 10:10 PM, Jed Brown wrote:
>>>>> On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <bojan.niceno at psi.ch> wrote:
>>>>> No, I use global.
>>>>> 
>>>>> The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)
>>>>>  
>>>>> 
>>>>>   for(Int i=0; i<n; i++) {
>>>>>     Int gi = mesh.nodes[i].global_number;
>>>>>     VecGetValues(x, 1, &gi, &unk[i]);
>>>>>   }
>>>>> 
>>>>> "n" is defined as the number of cells inside, i.e. without buffers.  "unk" is my external array.  If I try to access buffer values, I use:
>>>>> 
>>>>>   for(Int i=0; i<N; i++) {
>>>>>     Int gi = mesh.nodes[i].global_number;
>>>>>     VecGetValues(x, 1, &gi, &unk[i]);
>>>>>   }
>>>>> 
>>>>> But then I end up with tons of warnings, presumably because I am going beyond "n".  Vector x was created with VecCreateGhost.
>>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> <Mail Attachment.png>
>>> 
>>> 
>>> -- 
>>> <Mail Attachment.png>
>>> 
>> 
>> 
>> -- 
>> <Mail Attachment.png>
>> 
> 
> 
> -- 
> <Signature.png>