[petsc-users] partial stencil in DMDA?

[Barry Smith]

   Greg,

     The general mechanism for moving elements of vector between processes is with VecScatterCreate() and VecScatterBegin/End() with it you can indicate exactly what values are to move and to where, but it is all based on “one dimension” indexing into the vector. 

    DMDA provides a subset of communication for “structured meshes” (essentially 1,2, 3 dimensional arrays split among processes with horizontal and vertical cuts). It is pretty much all or nothing in terms of communicating neighbor information. 

    VecCreateGhost() uses the VecScatter mechanism to set up a SINGLE communication pattern between a Vec and its ghosted partner Vec. 

   Based on your email: “However, when communicating ghost values, not all of those degrees of freedom should be exchanged at once.” it sounds like you need several (many) communication patterns requiring different ghost entries in each. 

1) The hard general case: if for each set of ghost points you may need several entries from one grid point and a different number of entries from another grid point VecScatterCreate() (called multiple times, one for each pattern) and VecScatterBegin/End() are intended for this purpose. However if you are working with a 2 dimension grid/array you want to do ghosting for you need to initially map your ghosting patterns from the 2d indexing to the 1d indexing of the VecScatterCreate() which is a bit painful. You can see the routine I mentioned before DMSetUp_DA_2D() but it would need to be heavily modified. 

2) The easy case: if, for example, you need just the first index from each point communicated as a ghost and then next the second and then next the third you can avoid all custom communication patterns and just create 2 DMDA, one with a dof of 100 (or what ever it is for your case) and one with dof of 1 then use VecStrideGather to pull out the one component of the global vector (with the dof of 100) into a Vec obtained from DMCreateGlobalVector() from the dof of 1 DMDA then use the DMGlobalToLocalBegin/End on the 1 dof DMDA and now you have your single ghost point at each point vector that you want.

   Barry




On Feb 21, 2014, at 1:12 PM, Fischer, Greg A. <fischega at westinghouse.com> wrote:

> Barry,
>  
> Thanks!  I have another question. The user manual says:
>  
>                 PETSc currently provides no container for multiple arrays sharing the same distributed array communication; note, however, that the dof parameter handles many cases of interest.
>  
> In my application, each space location will have on the order of hundreds of values associated with it (which I believe translates to dof=O(100) – I don’t see the “degrees of freedom” explicitly defined anywhere). However, when communicating ghost values, not all of those degrees of freedom should be exchanged at once. I need to be able to exchange one at a time.
>  
> It sounds like what I may want to do is use VecCreateGhost(), which would allow me to define exactly where the ghost points are, and then duplicate that vector using VecDuplicateVecs() for each DOF. I can then scatter the vectors individually as the need arises. Does that sound reasonable?
>  
> Greg
>  
> >-----Original Message-----
> >From: Barry Smith [mailto:bsmith at mcs.anl.gov]
> >Sent: Friday, February 21, 2014 12:21 PM
> >To: Fischer, Greg A.
> >Cc: petsc-users at mcs.anl.gov
> >Subject: Re: [petsc-users] partial stencil in DMDA?
> > 
> > 
> >On Feb 21, 2014, at 11:02 AM, Fischer, Greg A. <fischega at westinghouse.com>
> >wrote:
> > 
> >> Hello,
> >> 
> >> I’m interested in using PETSc to manage distributed arrays. Based on my
> >reading about the DMDA objects, I see that ghost points can be
> >communicated in box-type stencils or star-type stencils.
> >> 
> >> For my application, assume that I have a 2D DMDA object with star-type
> >stencils. For a typical local calculation, I only need to access ghost values
> >from two of the four directions at a time. For example, I’d like access to ghost
> >values in the South and East directions, but not in the North or West
> >directions. Communicating North and West data would seem to be wasting
> >bandwidth. Is there any way to accomplish this?
> > 
> >   Greg,
> > 
> >    There is not anything built in. Here is what I suggest:
> > 
> >1)  write your application code not worrying about the fact that the
> >DMGlobalToLocalBegin/End() is moving values you don’t need.
> > 
> >2) when your code is running correctly for your problem and giving useful
> >results if the communication times are impacting how long it takes to run you
> >can provide a custom communication pattern. It would involve little
> >additional coding essentially taking DMSetUp_DA_2D() which creates the list
> >of ghost points and removing the  unneeded ghost points.  But it would be
> >premature to do this optimization until you have a full working application
> >code.
> > 
> >   Barry
> > 
> >> 
> >> Thanks,
> >> Greg
> > 
> >