[petsc-dev] ISRestoreNonlocalIS?

[Jed Brown]

On Thu, Nov 18, 2010 at 22:50, Dmitry Karpeev <karpeev at mcs.anl.gov> wrote:

> In general, I don't see how ISAllGather can be any better than
> ISGetTotalIndices:
> the operations underneath are essentially the same.
> ISGetTotalIndices can take advantage of any special structure just as
> ISAllGather could:
> by specializing the implementation to those subtypes. This can save
> some communication
> (e.g., ISAllGather_Stride might need to gather the initial offsets,
> strides and the total sizes
>  only).  The minute the actual indices are requested from the
> resulting IS, it will have to allocate
> the same amount of storage that ISGetTotalIndices requires.
>

Indeed, but there are a lot of operations you can do on an IS without
getting the indices.  Also, gathering strides is O(# procs) instead of
O(global problem size), so there is a big communication difference.  I'm not
saying the difference will be immediately realized for the important use
cases, but one leaves a straightforward way to optimize, the other offers no
such thing.


> ISAllGather followed by ISDifference is NOT what I need from
> ISGetNonlocalIndices.
> That would eliminate the indices in the range of the local portion of
> the IS, which might
> also wipe out some of the remote indices just gathered: multiple ranks
> might contain the same
> global index.  Furthermore, even if all of the local ranges within the
> IS are disjoint,
> ISDifference is a relatively complicated operation, while I can easily
> excise the local
> part after the MPI_Allgatherv occurred, because, as a byproduct, I
> obtain the offset
> of the local part in the gathered indices. Otherwise I'd have to
> reconstruct it using a scan.
>

Okay, but what is the meaning of a parallel IS where a given global index
shows up in multiple procs' owned parts?  In particular when do you need the
"nonlocal indices" of such an IS?  I guess the answer is that it would be
needed to produce a "blown up" global matrix where the local pieces were
overlapping.  I know I mentioned this as a dirty hack for building
overlapping matrices on subcomms, but the thing you are actually looking for
is a matrix on a subcomm, where there are no duplicate indices in the IS of
the subcomm.  If you are doing that by making MatGetSubMatrices() handle
(isrow,iscol) on subcomms, then I think you would get the right thing with
ISDifference.  Note that ISDifference can be implemented very efficiently if
the arrays are sorted (I feel like I wrote code for that, or something very
similar, not too long ago.)

Jed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20101118/5128bf98/attachment.html>