[petsc-users] MatGetSubMatrix and memory usage

[Jed Brown]

On Wed, 23 Jun 2010 11:45:56 +0200, francois pacull <fpacull at fluorem.com> wrote:
> Dear PETSc team,
> 
> I have a question regarding the MatGetSubMatrix routine (release 
> 3.0.0-p7). I am using it to change the partitioning of a MATMPIAIJ matrix:

How many processes are you using to provide the numbers below?

The MPIAIJ implementation does a non-scalable operation ISAllGather and
caches the result so that all future calls (with MAT_REUSE_MATRIX) will
be fast.  If you won't be doing the operation again, you could destroy
this cache (it is PetscObjectCompose'd with the submatrix under the name
"ISAllGather"), but it won't change the peak usage since that index set
is needed when extracting the submatrix.

It is possible to write a scalable MatGetSubMatrix, but it would be
significantly more complicated than the current implementation and
hasn't been done.  Is this issue coming up with your use of
PCFieldSplit?  In that case, I would suggest living with the memory use
until you find a split that you like, then create a MatShell that holds
the blocks separately and implement MatGetSubMatrix (by matching against
your distributed index sets).  Dmitry is working on a new matrix type in
petsc-dev that may be able to do this for you, but it's not ready yet.

Jed