[petsc-users] SAMRAI and Petsc

[Boyce Griffith]

On 4/27/12 4:22 PM, Pitts, Todd Alan wrote:
> I have been working with SAMRAI and Petsc to develop an implicit
> hyperbolic solver. Both packages seem very well done -- I am new to
> both, however. The basic physics problem is the solution to the linear
> wave equation (or rather something very close to it). I am using SAMRAI
> to track the mesh and Petsc to solve the resulting implicit time
> stepping problem. I am giving the MPIcomm object from SAMRAI to Petsc
> but have noticed that SAMRAI distributes the processors across the
> domain seemingly randomly. Petsc on the other hand distributes them

SAMRAI shouldn't be *randomly* assigning patches to processors.  If you 
are using SAMRAI 3.X, the default behavior of the 
ChopAndPackLoadBalancer is to use a space-filling curve algorithm to 
assign patches to processors.  (I am not sure exactly how the other load 
balancer class implemented in SAMRAI, TreeLoadBalancer, works.)  If you 
are using SAMRAI 2.X, the default behavior is again to use a 
space-filling curve algorithm.  You can also setup your own parallel 
distribution algorithm if you are so inclined.

> sequentially for a matrix. In other words, SAMRAI might give processors
> with adjacent rank numbers nonadjacent patches while Petsc always
> assigns the FIRST n rows to the rank 0 process (where n is the number of
> rows requested by the rank 0 process), the NEXT m rows to the rank 1
> process, etc. This means that a nice, diagonally banded matrix is no
> longer banded (because we have reordered the rows according to the
> SAMRAI domain decomposition). Naturally, the solution to the linear
> problem is the same. However, I am concerned that I have lost the
> apparent diagonal structure of the matrix. Is this a problem for the
> linear solvers? I know that Petsc is used for implicit problems together
> with various meshing codes -- this is a common thing. I would like to
> use the library in its intended fashion -- is the proper thing to do to
> simply fill the rows you own and forget the fact that the structure is
> no longer banded? Am I missing something?

Another option is to use the SAMRAIVectorReal classes to provide a 
vector interface to native SAMRAI data, and to pass those to PETSc using 
a wrapper class.  This allows you to avoid having to copy data back and 
forth between SAMRAI and PETSc, although for linear solvers you will 
need to use shell matrices and preconditioners.  Of course, the big 
drawback of this approach is that you have to roll your own 
preconditioner implementations, which can be a lot of work.

There is some code to facilitate this in the SAMRAI distribution, but 
I've always found it better to maintain my own versions of these 
wrappers (there is some relevant code available from 
https://ibamr.googlecode.com).  I think Bobby Philip maintains similar 
wrappers, although I don't know whether they are readily available.

-- Boyce