[petsc-users] DMDA Question

[Ethan Coon]

On Fri, 2011-12-02 at 14:08 -0500, Tim Gallagher wrote:
> Hi,
> 
> I'm trying to create some grids for a finite volume simulation and am a little stuck on the best way to tackle it. The approach now is to use DMDACreate3d with known number of I, J, K points and PETSC_DECIDE for the processors. Then I am trying to create another DMDA with the same distribution to store the cell centers and actual solution vector. The problem with is is the following:
> 
> Consider 9 points and two processors. Processor one gets points [1,5] with point 6 as a ghost and processor two gets points [6,9] with point 5 as a ghost. But now to create/store cell centers, processor two needs point 4 to construct a cell center for it's ghost. 
> 
> I can certainly fetch that point and do the calculation, but I feel like there is a more elegant way to do this out there. 

There isn't really a more elegant way out there (as far as I know).  The usual approach for this is to do as you have suggested, with the following alteration -- information about a cell should only be calculated by the process that owns the cell.  So the approach would be:

- set the coordinates of the DA as normal via DMDASetCoordinates()
- call DMDAGetCoordinateDA(), get a local vec based upon that DA, and call DMGlobalToLocalBegin/End() to scatter from the (global) coordinate vector into your new local vec.
- then the cell 4-5 center is calculated by process 0 (who owns node 4 and has node 5 as a ghost), set the cell coordinates
- reverse the process if needed, calling DMDAGetCoordinateDA() on the cell DA, and DMGlobalToLocalBegin/End() to get cell "4-5"'s coordinates on process 1 (where it is a ghost cell). 

Ethan


> Has anybody used a DMDA to create a dual DMDA? 
> 
> Any advice would be appreciated,
> 
> Tim

-- 
------------------------------------
Ethan Coon
Post-Doctoral Researcher
Applied Mathematics - T-5
Los Alamos National Laboratory
505-665-8289

http://www.ldeo.columbia.edu/~ecoon/
------------------------------------