[petsc-users] Structured Finite Difference Method With Periodic BC Help

[Paul Urbanczyk]

Hello,

I'm trying to implement a CFD code using finite differences on a 
structured mesh with PETSc.

I'm having a bit of trouble understanding how to properly use the 
periodic boundary condition with distributed arrays, and need some 
clarification.

If I set the boundaries in the x-direction to be DM_BOUNDARY_PERIODIC, 
and set a stencil width of two, there should be two ghost cells in the 
x-direction at each end of the mesh, which looks like it's happening 
just fine.

However, it seems that the assumption being made by PETSc when filling 
in these values is that the mesh is a cell-centered finite difference, 
rather than a vertex-centered finite difference. Is there a way to 
change this behavior? In other words, I need the first ghost cell on 
each side to correspond to the opposite side's first interior point, 
rather than the opposite boundary point.

If there is not a way to change this behavior, then I need to set my own 
ghost cells; however, I'm having trouble implementing this...

If I change the boundaries to DM_BOUNDARY_GHOSTED, with a stencil width 
of two, I have the necessary ghost cells at either end of the mesh. I 
then try to do the following:

1) Scatter the global vector to local vectors on each rank

2) Get a local array referencing the local vectors

3) Calculate ghost values and fill them in the appropriate local arrays

4) Restore the local vectors from the arrays

5) Scatter the local vector info back to the global vector

However, if I then re-scatter and look at the local vectors, the ghost 
cell values are zero. It seems as though the ghost values are lost when 
scattered back to the global vector.

What am I doing wrong here?

Thanks for your help!

-Paul