[petsc-dev] Integrating PFLOTRAN, PETSC & SAMRAI

[Philip, Bobby]

________________________________________
From: petsc-dev-bounces at mcs.anl.gov [petsc-dev-bounces at mcs.anl.gov] On Behalf Of Jed Brown [jed at 59A2.org]
Sent: Tuesday, June 07, 2011 2:40 PM
To: For users of the development version of PETSc
Subject: Re: [petsc-dev] Integrating PFLOTRAN, PETSC & SAMRAI

On Tue, Jun 7, 2011 at 20:24, Philip, Bobby <philipb at ornl.gov> wrote:

> The overhead of constructing a native PETSc vector across all refinement
> levels would involve writing a global ordering
> scheme that numbers nodes/cells/edges etc (depending on the data centering)
> and then doing maps back and forth. For my larger simulations
> the cost for this would be too high because this would not be a one time
> cost.
>

Did you profile this? The cost seems very dependent on the implementation.
The fundamental part is basically a copy from contiguous to discontiguous
data. That memcpy for each patch might not be very expensive compared to
e.g. the traversal.

The issue is that for a given patch only a subset of the cells which do not have
finer refinement cells would do the memcpy. This would be a very irregular data
operation. I have not profiled this as I would first have to write the global ordering
etc.

Is this whole hierarchy manipulated directly by the user, or is there an
iterator for patches? Could an iterator be made to extract the patch with
ghosts from a contiguous PETSc vector?

Again, the subset of cells only being a part of the PETSc vector would be the
issue that is complicating.

> I would have to do this each time PETSc KSP made a call to
> one of my multilevel preconditioners such as FAC.
>

It seems to me that setup should only need to happen when the mesh adapts.
Are you adapting the mesh inside of a KSPSolve? (I think that would be
strange.)

Every apply call from KSP does need to initialize data over the whole
hierarchy that is consistent across the levels.