[petsc-dev] ISRestoreNonlocalIS?

[Barry Smith]

On Nov 18, 2010, at 3:53 PM, Jed Brown wrote:

> On Thu, Nov 18, 2010 at 22:33, Barry Smith <bsmith at mcs.anl.gov> wrote:
> We could bag the in-place  ISSort() and make it return a new IS.
> 
> Sounds good to me.
> 
> > Could we at least get rid of ISGetTotalIndices() and ISGetNonlocalIndices() in favor of ISGetNonlocalIS() followed by ISGetIndices(), so that the interface allows working with ISStride and ISBlock efficiently?
> 
>  The problem is what is the "meaning" of that new IS? What communicator does it live on? It is returning information about a given IS, it really isn't an IS at all given my definition of IS from the previous email.  If you could come up with proper model for this beast (and have it cached in the original IS) then I would be happy if it was an IS and we didn't need the GetNonlocalIndices() and GetAllIndices().  With ISAllGather() it returns a sequential IS that is then passed in from MatGetSubMatrix_MPIAIJ() to MatGetSubMatrix_MPIAIJ_Private() to MatGetSubMatrices() but now you are passing a sequential IS to MatGetSubMatrices() while the isrow may be (or is) a parallel IS.  Explain this to me and we'll have a good model.
> 
> I've chatted with Dmitry about this and I think that MatGetSubMatrices should produce matrices on the communicator pair (isrow,iscol).

   Jed, this is what started this entire process that you are unhappy with :-).


>  Currently it only produces serial matrices, in which case both isrow and iscol should be serial.  But to extract on subcomms, the user would pass in (isrow,iscol) living on subcomms.  This would actually make MatGetSubMatrix a redundant API because the user could pass a single global (isrow,iscol) to MatGetSubMatrices().  Along these lines, it is easy to implement the trivially cheap ISGetLocalIS() so that, e.g. MatGetSubMatrix_MPIAIJ_Private doesn't need to do anything weird.

   We are getting closer.   But you still haven't resolved my big questions.

(1)  Where do we cache ISGetAllIndices(IS,IS*) (replacement for ISAllGather()) and ISGetNonlocalIndices(IS,IS*)? I want to cache them in the original IS, is that ok with you?

(2) What is the meaning of the IS that is returned? It really bothers me that it is a sequential beast. What is it suppose to mean? To me it is information about a parallel object therefor belongs in that parallel object.  Am I just wrong to be bothered by it?

> 
> An IS packs a lot of semantic information by having a communicator, some of the current interfaces ignore this for historical reasons.

   True. For many years it wasn't clear, what, if anything the communicator in the IS meant.

   Barry



>  
>   In the past we did not always wrap "collections of indices" as an IS, I think it still exists from that time. If it is never used and will not be needed it could be removed.
> 
> Done.
> 
> Jed