[petsc-users] Repacking/scattering of multi-dimensional arrays

[Jed Brown]

Matt, Jack, and myself have discussed adding proper support for Elemental
with PETSc, but we just haven't had the time. Matt and I don't have a
direct use case and we all have plenty of other things going.

On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn <
d.s.seljebotn at astro.uio.no> wrote:

> Hi list,
>
> I'm wondering whether there's any code in PETSc that I can either use
> directly or lift out and adapt for my purposes (which are described in
> footnote [1]).
>
> What I need to do is a number of different 2D array ("multi-vector",
> "dense matrix") repacking operations, in order to make the data available
> to different operations (spherical harmonic transforms, dense linear
> algebra, sparse linear algebra, etc.).
>
> There's a number of repacking operations I'm in need of:
>
>  - From each process having a given contiguous set of rows of a 2D array,
> to an element-cyclic distribution where one distributes over both rows and
> columns (the distribution of Elemental [2])
>
>  - Arbitrary redistribution of rows between processes (but each row is
> kept in a process), potentially with overlaps.
>
>  - More structured redistribution of rows between processes where indices
> can be easily computed on the fly (but still a pattern quite specific to my
> application).
>
> While I found PETSc routines for arbitrary redistribution of 1D arrays, I
> couldn't find anything for 2D arrays. Any pointers into documentation or
> papers etc. explaining these aspects of PETSc is very welcome.
>

It sounds like you just want help with indexing. VecScatter can move data
of any dimensionality, you just have to know how to index it. (You could
ask for something more specialized if you were using the topology routines
in MPI and wanted control over how messages are routed.)


>
> What I'd *really* like (and what I'll do if I end up doing it myself) is
> something more generic: A small, focused library which, given two
> descriptors of how an N-dimensional array is distributed, does the
> redistribution from one to the other. What I'd imagine is that you for each
> axis of the array pick an axis distribution ("chunked", "(block-)cyclic",
> "provided by indices", "non-distributed"), and an MPI datatype. (The array
> would have to be distributed on a process grid of the same dimension as the
> array).
>
> Then, given two N-dimensions descriptors which mixes structured and
> unstructured information, the library can make and execute a redistribution
> plan.
>
> Perhaps too ambitious, which is why I'm hoping that somebody can point me
> to at least something existing. I can imagine doing most of it from
> scratch, but reusing PETSc for the unstructured axis-distribution type.
>
> Comments of any kind welcome.
>
> Dag
>
> [1] What I'm doing is solving a linear system by CG where the LHS matrix
> can only be written as a (rather large) symbolic equation, containing lots
> of spherical harmonic transforms, Fourier transform, both dense and sparse
> linear algebra, and so on.
>
> [2] http://code.google.com/p/**elemental/<http://code.google.com/p/elemental/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120326/1ca8ce06/attachment.htm>