[petsc-dev] ISRestoreNonlocalIS?

[Barry Smith]

On Nov 18, 2010, at 4:14 PM, Jed Brown wrote:

> On Thu, Nov 18, 2010 at 23:04, Barry Smith <bsmith at mcs.anl.gov> wrote:
> Jed, this is what started this entire process that you are unhappy with :-).
> 
> I know, but I was bothered by an IS interface that destroys structure (necessarily flattens everything to scalar indices).

   I understand your concern.
> 
> >  Currently it only produces serial matrices, in which case both isrow and iscol should be serial.  But to extract on subcomms, the user would pass in (isrow,iscol) living on subcomms.  This would actually make MatGetSubMatrix a redundant API because the user could pass a single global (isrow,iscol) to MatGetSubMatrices().  Along these lines, it is easy to implement the trivially cheap ISGetLocalIS() so that, e.g. MatGetSubMatrix_MPIAIJ_Private doesn't need to do anything weird.
> 
>   We are getting closer.   But you still haven't resolved my big questions.
> 
> (1)  Where do we cache ISGetAllIndices(IS,IS*) (replacement for ISAllGather()) and ISGetNonlocalIndices(IS,IS*)? I want to cache them in the original IS, is that ok with you?
> 
> Yes, perhaps with some way to free that memory later (we've already gotten questions about why calling MatGetSubMatrix triples  the memory usage of the original matrix, and they have to make another copy, and then destroy the matrix they used to get the submatrix in order to reclaim that memory).  But that can be a long-term API issue.
> 
> If IS is immutable, the I definitely agree with caching the gathered part on the IS.  But MatGetSubMatrix_MPIAIJ should also cache the original IS so that it doesn't produce garbage if the user tries to MAT_REUSE_MATRIX with a different IS.

   I have no problem with the new matrix also caching the ORIGINAL IS for a quick check. That's the particular implementations business.

>  
> (2) What is the meaning of the IS that is returned? It really bothers me that it is a sequential beast. What is it suppose to mean? To me it is information about a parallel object therefor belongs in that parallel object.  Am I just wrong to be bothered by it?
> 
> It doesn't bother me that it is sequential.  After all, you are requesting a sequential matrix defined by extracting the part defined by sequential (isrow,iscol) from the parallel matrix.

   In this particular case. But will this be true for all uses of ISAllGather() (or any new name given it) in the future?

   Barry


>  Other procs are requesting different parts and will get back their own sequential parts.
>  
> Jed