[petsc-dev] Vector operations on ghosted vectors

[Barry Smith]

On Mar 2, 2013, at 7:58 AM, John Mousel <elafint.john at gmail.com> wrote:

> Matt,
> 
> Please correct the place where my logic breaks down. I must be misunderstanding how something works.
> 
> 1. VecAXPY -> sums locally owned portions of vector; ghosts are not summed.
> 2. Scatter ghosts of summed vector because they were not touched by VecAXPY
> 
> 
> It was my belief that if I called VecAXPY on a ghosted vector without calling VecGhostGetLocalForm, it would only sum the local entries. If I call VecGhostGetLocalForm and perform VecAXPY on the resulting vectors, it will sum the ghosts as well. Your response makes me think I've got this wrong.
> 
> Sorry about whatever misunderstanding is occuring. I do appreciate your help.


  John,

   Your understanding is correct.   Look, just write your wrappers and be happy, don't troll Matt and Jed :-)


   Barry

> 
> John
> 
> 
> 
> On Sat, Mar 2, 2013 at 7:06 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Mar 1, 2013 at 12:40 PM, John Mousel <elafint.john at gmail.com> wrote:
> So am I right in reading this as you suggesting to: 
> 
> 1. Compute VecAXPY on all the u(n),u(n-1)...locally
> 2. Scatter the result of step (1)
> 
> Why is this here? You don't need that at all since the local space is "ghosted".
> 
>     Matt
>  
> 3. Perform the divergence on the resulting ghosted vector from (2).
> 
> I don't see how that is either cleaner or more efficient than just summing up the ghosted portions of the vector since they have not been altered since the solve at the previous time step.
> I hope I'm being clear that what I am proposing is just redundant computation to avoid communication?
> 
> 
> On Fri, Mar 1, 2013 at 11:31 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Mar 1, 2013 at 10:37 AM, John Mousel <elafint.john at gmail.com> wrote:
> Let me be even more specific. I'm implementing a velocity correction scheme in rotational form (reference given below). In the first step, forgetting even a sub-iterative loop, a projecting pressure is solved for from 
> 
> L(p(n+1)) = -div(beta*u(n) + gamma*u(n-1) + exci*u(n-2) + H(n+1) - H(n) + f(n+1) - f(n))
> 
> Here, L and div are Laplacian and divergence operators, and H(n) and f(n) are the non-linear and body force at time level n. In my mind, to form part of the RHS, I can use VecAXPY on all the terms in the div(), and then perform the divergence on that. To avoid communication, I can just use the VecAXPY on the ghosted portion as well, which since I've copied the entire ghosted vector during the VecCopy(u(n),u(n-1)) operation, should make sense.
> Is that not a wise thing to do?
> 
> You have some communication of the ghosts before the solve. Why not work completely in the local space
> for this assembly, and then completely in the global space for the solve? We usually organize this way.
> 
>   Matt
>  
> John
> 
> J. L. Guermond, P. Minev, and J. Shen. An overview of projection methods for incompressible flows. Computer Methods in Applied Mechanics and Engineering, 195(44-47):6011–6045, 2006.
> 
> 
> On Fri, Mar 1, 2013 at 9:28 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Fri, Mar 1, 2013 at 10:24 AM, John Mousel <elafint.john at gmail.com> wrote:
> I use the data in the vectors for things besides pure vector operations. In the context of a sub-iteration scheme for an incompressible solver, I want to use the values stored in the vector at the previous iteration to construction the non-linear term. This requires copying not just the local portion of the vector in the u(k)->u(k-1) operation, but the ghosts as well. Doesn't copying the entire ghosted vector make sense in this context?
> 
> It seems like a strange work flow. Since none of the vector operations touch these, you will be getting
> stale values anyway. It would make more sense to me if you were using GlobalToLocal() during the iteration.
> 
>    Matt
>  
> John
> 
> 
> On Fri, Mar 1, 2013 at 8:09 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
> On Wed, Feb 27, 2013 at 4:56 PM, John Mousel <john.mousel at gmail.com> wrote:
> Is there a possibility of adding a wrapper function around a few basic vector operations such as VecCopy, VecAXPY, VECAXPYPZ... to operate on ghosted vectors? I perform a lot of vector operations including the ghost region to avoid communication.
> 
> Can you explain the context where operating with local forms in this way makes sense? Usually you would either be working locally, in which case you copy between local forms (or purely local work vectors) or you are operating globally in which case there is nothing to gain by applying operations to the local form.
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>