[petsc-dev] ISRestoreNonlocalIS?

[Dmitry Karpeev]

I'm on the fence about the high level considerations of API, the
semantics of ISs, etc,
but here are some technical comments.

On Thu, Nov 18, 2010 at 2:15 PM, Jed Brown <jed at 59a2.org> wrote:
> What is going on here?  If IS is caching this nonlocal stuff, then every
> function causing mutation (e.g. ISGeneralSetIndices, and a few others) needs
> to be visited and made to discard the gathered and nonlocal caches.

It is generally thought (at least by Barry :-)) that ISs should be
thought of as immutable
objects.  In any event, it is easy to mark the cache as invalid upon
an invocation of ISGeneralSetIndices
or similar mutation operations by ISStrides.

> But I
> think this whole idea is just dangerous and abuses the IS.  The user can
> always do ISAllGather if they really want everything, and it's potentially
> cheaper than this new ISGetTotalIndices because it can maintain ISStride or
> ISBlock structure.

In general, I don't see how ISAllGather can be any better than
ISGetTotalIndices:
the operations underneath are essentially the same.
ISGetTotalIndices can take advantage of any special structure just as
ISAllGather could:
by specializing the implementation to those subtypes. This can save
some communication
(e.g., ISAllGather_Stride might need to gather the initial offsets,
strides and the total sizes
 only).  The minute the actual indices are requested from the
resulting IS, it will have to allocate
the same amount of storage that ISGetTotalIndices requires.

> If you want all the indices, use ISAllGather and then
> ISGetIndices, it will cost the same and then the user has explicit control
> over when to free this memory.  In principle, ISGetNonlocalIS could be
> implemented to avoid the intermediate storage of ISAllGather followed by
> ISDifference, but it's not currently done this way and currently both copies
> are cached so they have to both exist, and since the IS is managing
> ownership of this other nonlocal IS, you can't keep the nonlocal part.

ISAllGather followed by ISDifference is NOT what I need from
ISGetNonlocalIndices.
That would eliminate the indices in the range of the local portion of
the IS, which might
also wipe out some of the remote indices just gathered: multiple ranks
might contain the same
global index.  Furthermore, even if all of the local ranges within the
IS are disjoint,
ISDifference is a relatively complicated operation, while I can easily
excise the local
part after the MPI_Allgatherv occurred, because, as a byproduct, I
obtain the offset
of the local part in the gathered indices. Otherwise I'd have to
reconstruct it using a scan.


> I just don't understand the point of all these new functions, they feel like
> an inferior interface to something that is currently available.
> Same question for ISAllGatherIndices which has been around for a long time.
>  It looks to me like the user can do ISCreateGeneral(), ISAllGather(),
> ISGetIndices() to achieve exactly the same thing (no performance
> difference).  Why even call it IS when no IS (or PETSc object of any kind is
> used by this function)?  It is never used in PETSc.

I totally agree with that last point.
As far as ISGetTotalIS, etc, I needed to wrap the results of gathering
the nonlocal indices
(with and without the local part) as ISs that can be fed into
MatGetSubMatrices_XXX, which
only accepts ISs, not index arrays.

Dmitry.
>
> Jed