[petsc-users] Repacking/scattering of multi-dimensional arrays

[Dag Sverre Seljebotn]

On 03/26/2012 12:06 PM, Jed Brown wrote:
> Matt, Jack, and myself have discussed adding proper support for
> Elemental with PETSc, but we just haven't had the time. Matt and I don't
> have a direct use case and we all have plenty of other things going.
>
> On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn
> <d.s.seljebotn at astro.uio.no <mailto:d.s.seljebotn at astro.uio.no>> wrote:
>
>     Hi list,
>
>     I'm wondering whether there's any code in PETSc that I can either
>     use directly or lift out and adapt for my purposes (which are
>     described in footnote [1]).
>
>     What I need to do is a number of different 2D array ("multi-vector",
>     "dense matrix") repacking operations, in order to make the data
>     available to different operations (spherical harmonic transforms,
>     dense linear algebra, sparse linear algebra, etc.).
>
>     There's a number of repacking operations I'm in need of:
>
>       - From each process having a given contiguous set of rows of a 2D
>     array, to an element-cyclic distribution where one distributes over
>     both rows and columns (the distribution of Elemental [2])
>
>       - Arbitrary redistribution of rows between processes (but each row
>     is kept in a process), potentially with overlaps.
>
>       - More structured redistribution of rows between processes where
>     indices can be easily computed on the fly (but still a pattern quite
>     specific to my application).
>
>     While I found PETSc routines for arbitrary redistribution of 1D
>     arrays, I couldn't find anything for 2D arrays. Any pointers into
>     documentation or papers etc. explaining these aspects of PETSc is
>     very welcome.
>
>
> It sounds like you just want help with indexing. VecScatter can move
> data of any dimensionality, you just have to know how to index it. (You
> could ask for something more specialized if you were using the topology
> routines in MPI and wanted control over how messages are routed.)

Thanks for your response. Now I know what question to ask:

The problem with VecScatter is that it requires indexing all elements, 
which can be rather wasteful in some situations. So I'm wondering if 
VecScatter could potentially work with elements of any size, so that one 
could send a bunch of elements (a picture of this is only indexing rows 
of a row-major 2D array, thus amortizing the indexing overhead and 
memory use by the number of columns).

(That doesn't cover the Elemental case, where one would need to index 
every element (rather wasteful), but it does cover some of my other needs).

Dag