[petsc-dev] Extract sub-matrices efficiently

[Barry Smith]

  Are the many small matrices you are extracting consisting of parts coming from different processes? Or does each sub matrix come from the same process (and remain on the same process) or are most coming from the same process but some shared across multiple processes? 

   The parallel code was originally written assuming one wishes to get a small number of large sub matrices. Thus it is a bit communication heavy, for example it doesn't share any communications or all reduce across the matrices. It is, of course, possible to write custom code for your case but in order to help you to do that we need to know more about your particular use case.

   Barry


> On May 18, 2016, at 11:41 AM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Could you give some more information about your preconditioner?
> Sequential extraction of many small submatrices is not high performance,
> but perhaps we can restructure.
> 
> victor.magri at dicea.unipd.it writes:
> 
>> 
>> 
>> Dear PETSc developers, 
>> I have a matrix of the type MATMPIAIJ from which I want to extract small
>> sub-matrices and sub-vectors (number of rows from 1 to 50 maximum)
>> several times. This is a very fundamental step of a preconditioner that
>> I'm writing. After that, I want to solve these sub-systems locally. 
>> 
>> In my current implementation I use MatGetSubMatrices with
>> MAT_INITIAL_MATRIX to extract the sub-matrices and MatGetRow plus some
>> other artifacts to extract the sub-vectors. However, I see that this
>> procedure is too costly, once it creates and destroys the output
>> variables each time they are called. I thought about using the MAT_REUSE
>> flag, but I can't since the nonzero pattern of the local sub-matrix may
>> change from call to call. 
>> 
>> Can you give me some hint on how to do this task efficiently? I
>> appreciate your help! 
>> 
>> -- 
>> 
>> Victor A. P. Magri - PhD student
>> Dept. of Civil, Environmental and Architectural Eng.
>> University of Padova
>> Via Marzolo, 9 - 35131 Padova, Italy