[petsc-users] SAMRAI and Petsc

[Barry Smith]

  Todd,

    The specific diagonally banded structure is not something that can usually be taken advantage of in parallel, so don't worry about it not have this exact structure.

    However, the fact that SAMRAI is "randomly" assigning data chunks to processes seems like it will be problematic for large core counts (1000s of cores) since the ghost point updates may be slower; but this is really a SAMRAI only issue and not related to PETSc. So the question is how does SAMRAI decide where to assign chunks and is there any way to improve that. Looking its documentation and asking the SAMRAI developers about this issue may be useful.

   Barry

On Apr 27, 2012, at 3:22 PM, Pitts, Todd Alan wrote:

> All,
> 
> I have been working with SAMRAI and Petsc to develop an implicit hyperbolic solver.  Both packages seem very well done -- I am new to both, however.  The basic physics problem is the solution to the linear wave equation  (or rather something very close to it).  I am using SAMRAI to track the mesh and Petsc to solve the resulting implicit time stepping problem.  I am giving the MPIcomm object from SAMRAI to Petsc but have noticed that SAMRAI distributes the processors across the domain seemingly randomly.  Petsc on the other hand distributes them sequentially for a matrix.  In other words, SAMRAI might give processors with adjacent rank numbers nonadjacent patches while Petsc always assigns the FIRST n rows to the rank 0 process (where n is the number of rows requested by the rank 0 process), the NEXT m rows to the rank 1 process, etc.  This means that a nice, diagonally banded matrix is no longer banded  (because we have reordered the rows according to the SAMRAI domain decomposition).  Naturally, the solution to the linear problem is the same.  However, I am concerned that I have lost the apparent diagonal structure of the matrix.   Is this a problem for the linear solvers?  I know that Petsc is used for implicit problems together with various meshing codes -- this is a common thing.  I would like to use the library in its intended fashion -- is the proper thing to do to simply fill the rows you own and forget the fact that the structure is no longer banded?  Am I missing something?
> 
> Thanks in advance for any help you might offer,
> 
> Todd
>