[petsc-users] MatGetSubMatrix and memory usage

[francois pacull]

Thanks Jed for your quick and helpful answer!

On 23/06/2010 12:31, Jed Brown wrote:
> On Wed, 23 Jun 2010 11:45:56 +0200, francois pacull<fpacull at fluorem.com>  wrote:
>    
>> Dear PETSc team,
>>
>> I have a question regarding the MatGetSubMatrix routine (release
>> 3.0.0-p7). I am using it to change the partitioning of a MATMPIAIJ matrix:
>>      
> How many processes are you using to provide the numbers below?
>    
I used only 2 processes in that test.
> The MPIAIJ implementation does a non-scalable operation ISAllGather and
> caches the result so that all future calls (with MAT_REUSE_MATRIX) will
> be fast.  If you won't be doing the operation again, you could destroy
> this cache (it is PetscObjectCompose'd with the submatrix under the name
> "ISAllGather"), but it won't change the peak usage since that index set
> is needed when extracting the submatrix.
>
> It is possible to write a scalable MatGetSubMatrix, but it would be
> significantly more complicated than the current implementation and
> hasn't been done.  Is this issue coming up with your use of
> PCFieldSplit?
No, I was just trying to understand the results I got from a call to 
PetscMemoryGetMaximumUsage. I am dealing with a very large matrix and 
trying to save memory...
> In that case, I would suggest living with the memory use
> until you find a split that you like, then create a MatShell that holds
> the blocks separately and implement MatGetSubMatrix (by matching against
> your distributed index sets).  Dmitry is working on a new matrix type in
> petsc-dev that may be able to do this for you, but it's not ready yet.
>
> Jed
>
>    
francois.