<div dir="ltr">Actually now that we are at it, what would you guys recommend as the best way to find local indecies from global ones when accessing the _ghost_ values in the local form? This becomes even more complicated when your global petsc ordering is different from global application ordering.<div>
<br></div><div>Its quite easy to access the local indecies for the local nodes -- you use ao to convert application global to petsc global and from petsc global, it is easy to get petsc local. However, when you have ghost nodes, there is no such simple relationship between petsc global and the _ghosted_ petsc local (Even though there is a simple one from petsc local to petsc global). The best thing I have came up with so far is to use a sequential second ao to map local indecies of ghost points to petsc global indecies. Is there a better way of doing this?</div>
<div><br></div><div>Mohammad<br><br><div class="gmail_quote">On Thu, Feb 23, 2012 at 7:44 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I have added the following to the VecGhostGetLocalForm() manual page to make this clearer for everyone:<br>
<br>
To update the ghost values from the locations on the other processes one must call<br>
VecGhostUpdateBegin() and VecGhostUpdateEnd() before accessing the ghost values. Thus normal<br>
usage is<br>
$ VecGhostUpdateBegin(x,INSERT_VALUES,SCATTER_FORWARD);<br>
$ VecGhostUpdateEnd(x,INSERT_VALUES,SCATTER_FORWARD);<br>
$ VecGhostGetLocalForm(x,&xlocal);<br>
$ VecGetArray(xlocal,&xvalues);<br>
$ /* access the non-ghost values in locations xvalues[0:n-1] and ghost values in locations xvalues[n:n+nghost]; */<br>
$ VecRestoreArray(xlocal,&xvalues);<br>
$ VecGhostRestoreLocalForm(x,&xlocal);<br>
<div><div><br>
<br>
On Feb 23, 2012, at 5:38 PM, Bojan Niceno wrote:<br>
<br>
> Yeeee-ha!<br>
><br>
> VecGhostUpdateBegin() / VecGhostUpdateEnd() was indeed what I was missing.<br>
><br>
> Thank you Mohammad, thank you all who helped me today!<br>
><br>
><br>
> Cheers<br>
><br>
><br>
> Bojan<br>
><br>
><br>
><br>
> On 2/24/2012 12:27 AM, Mohammad Mirzadeh wrote:<br>
>> You also need calls to VecGhostUpdateBegin()/VecGhostUpdateEnd() functions to update the ghost values if you change them in the global representation. See Petsc Manual 3.2 pp 55-56<br>
>><br>
>> Mohammad<br>
>><br>
>> On Thu, Feb 23, 2012 at 3:18 PM, Bojan Niceno <<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>> wrote:<br>
>> On 2/23/2012 11:49 PM, Mohammad Mirzadeh wrote:<br>
>>> based on,<br>
>>><br>
>>> VecCreateGhost(PETSC_COMM_WORLD, n, PETSC_DECIDE, nghost, &ghosts[0], &x);<br>
>>><br>
>>><br>
>>> it seems to me that x is your actual ghosted vector. If this is true, then you need to get its "local" form via VecGhostGetLocalForm(). Once you have done, you should be able to access the ghosted nodes. Are you calling this function anywhere?<br>
>><br>
>> I tried that before. I did:<br>
>><br>
>> Vec lx;<br>
>> VecGhostGetLocalForm(x, &lx)<br>
>><br>
>> then I copied "lx" to my variable, like<br>
>><br>
>> for(Int i=1; i<N; i++) /* N includes buffer cells */<br>
>> unk[i] = lx[i]<br>
>><br>
>> but ghost values were also zero. I am thinking that PETSc somehow clears the ghost values after a call to KSP. Is it the case?<br>
>><br>
>><br>
>> Kind regards,<br>
>><br>
>><br>
>> Bojan<br>
>><br>
>><br>
>>><br>
>>> On Thu, Feb 23, 2012 at 2:32 PM, Bojan Niceno <<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>> wrote:<br>
>>> Dear Mohammad,<br>
>>><br>
>>><br>
>>> it doesn't help me, or I did not understand your explanation.<br>
>>><br>
>>> If I do this:<br>
>>><br>
>>> /* copy internal values (THIS WORKS, BUT COPIES NO BUFFER VALUES) */<br>
>>><br>
>>> for(Int i=0; i<n; i++) {<br>
>>> Int gi = mesh.nodes[i].global_number;<br>
>>> VecGetValues(x, 1, &gi, &unk[i]);<br>
>>> }<br>
>>><br>
>>> /* copy ghost values (CREATES MANY WARNINGS */<br>
>>><br>
>>> for(Int i=n; i<N; i++) {<br>
>>> VecGetValues(x, 1, &i, &unk[i]);<br>
>>> }<br>
>>><br>
>>> I get arnings are like this.<br>
>>><br>
>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>> [0]PETSC ERROR: Argument out of range!<br>
>>> [0]PETSC ERROR: Can only get local values, trying 3518!<br>
>>> [3]PETSC ERROR: --------------------- Error Message ------------------------------------<br>
>>> [3]PETSC ERROR: Argument out of range!<br>
>>> [3]PETSC ERROR: Can only get local values, trying 3511!<br>
>>> [3]PETSC ERROR: ------------------------------------------------------------------------<br>
>>><br>
>>> What am I doing wrong here?<br>
>>><br>
>>><br>
>>> Cheers,<br>
>>><br>
>>><br>
>>> Bojan<br>
>>><br>
>>><br>
>>><br>
>>> On 2/23/2012 11:23 PM, Mohammad Mirzadeh wrote:<br>
>>>> just index x with the local numberings. if you have 'n' local nodes and 'g' ghost nodes in x, ghost nodes indecies run from 'n' to 'n+g-1'<br>
>>>><br>
>>>> On Feb 23, 2012 1:16 PM, "Bojan Niceno" <<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>> wrote:<br>
>>>> Dear Jed,<br>
>>>><br>
>>>> thanks.<br>
>>>><br>
>>>> Now I have the following:<br>
>>>><br>
>>>> - Array unk, which should hold values inside the partition and in ghost cells. It is big enough to hold both<br>
>>>> - Vec x, created by command VecCreateGhost, with proper padding for ghost cells<br>
>>>> - Successful call to linear solver in parallel.<br>
>>>><br>
>>>> But I need to copy ghost values from x to my array unk.<br>
>>>><br>
>>>> How can I do it?<br>
>>>><br>
>>>><br>
>>>> Kind regards,<br>
>>>><br>
>>>><br>
>>>> Bojan<br>
>>>><br>
>>>> On 2/23/2012 10:10 PM, Jed Brown wrote:<br>
>>>>> On Thu, Feb 23, 2012 at 15:05, Bojan Niceno <<a href="mailto:bojan.niceno@psi.ch" target="_blank">bojan.niceno@psi.ch</a>> wrote:<br>
>>>>> No, I use global.<br>
>>>>><br>
>>>>> The local form is just a local vector. It doesn't even know that a global problem exists. You can't index into it using global indices. (In general, there is no efficient way to look up information in the local vector (includes ghost points) using global indices.)<br>
>>>>><br>
>>>>><br>
>>>>> for(Int i=0; i<n; i++) {<br>
>>>>> Int gi = mesh.nodes[i].global_number;<br>
>>>>> VecGetValues(x, 1, &gi, &unk[i]);<br>
>>>>> }<br>
>>>>><br>
>>>>> "n" is defined as the number of cells inside, i.e. without buffers. "unk" is my external array. If I try to access buffer values, I use:<br>
>>>>><br>
>>>>> for(Int i=0; i<N; i++) {<br>
>>>>> Int gi = mesh.nodes[i].global_number;<br>
>>>>> VecGetValues(x, 1, &gi, &unk[i]);<br>
>>>>> }<br>
>>>>><br>
>>>>> But then I end up with tons of warnings, presumably because I am going beyond "n". Vector x was created with VecCreateGhost.<br>
>>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
</div></div>>>>> <Mail Attachment.png><br>
>>><br>
>>><br>
>>> --<br>
>>> <Mail Attachment.png><br>
>>><br>
>><br>
<span><font color="#888888">>><br>
>> --<br>
>> <Mail Attachment.png><br>
>><br>
><br>
><br>
> --<br>
> <Signature.png><br>
<br>
</font></span></blockquote></div><br></div></div>