[petsc-users] Repacking/scattering of multi-dimensional arrays

[Dag Sverre Seljebotn]

Hi list,

I'm wondering whether there's any code in PETSc that I can either use 
directly or lift out and adapt for my purposes (which are described in 
footnote [1]).

What I need to do is a number of different 2D array ("multi-vector", 
"dense matrix") repacking operations, in order to make the data 
available to different operations (spherical harmonic transforms, dense 
linear algebra, sparse linear algebra, etc.).

There's a number of repacking operations I'm in need of:

  - From each process having a given contiguous set of rows of a 2D 
array, to an element-cyclic distribution where one distributes over both 
rows and columns (the distribution of Elemental [2])

  - Arbitrary redistribution of rows between processes (but each row is 
kept in a process), potentially with overlaps.

  - More structured redistribution of rows between processes where 
indices can be easily computed on the fly (but still a pattern quite 
specific to my application).

While I found PETSc routines for arbitrary redistribution of 1D arrays, 
I couldn't find anything for 2D arrays. Any pointers into documentation 
or papers etc. explaining these aspects of PETSc is very welcome.

What I'd *really* like (and what I'll do if I end up doing it myself) is 
something more generic: A small, focused library which, given two 
descriptors of how an N-dimensional array is distributed, does the 
redistribution from one to the other. What I'd imagine is that you for 
each axis of the array pick an axis distribution ("chunked", 
"(block-)cyclic", "provided by indices", "non-distributed"), and an MPI 
datatype. (The array would have to be distributed on a process grid of 
the same dimension as the array).

Then, given two N-dimensions descriptors which mixes structured and 
unstructured information, the library can make and execute a 
redistribution plan.

Perhaps too ambitious, which is why I'm hoping that somebody can point 
me to at least something existing. I can imagine doing most of it from 
scratch, but reusing PETSc for the unstructured axis-distribution type.

Comments of any kind welcome.

Dag

[1] What I'm doing is solving a linear system by CG where the LHS matrix 
can only be written as a (rather large) symbolic equation, containing 
lots of spherical harmonic transforms, Fourier transform, both dense and 
sparse linear algebra, and so on.

[2] http://code.google.com/p/elemental/