Matt, Jack, and myself have discussed adding proper support for Elemental with PETSc, but we just haven't had the time. Matt and I don't have a direct use case and we all have plenty of other things going.<div><br>
<div class="gmail_quote">On Mon, Mar 26, 2012 at 13:53, Dag Sverre Seljebotn <span dir="ltr"><<a href="mailto:d.s.seljebotn@astro.uio.no">d.s.seljebotn@astro.uio.no</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi list,<br>
<br>
I'm wondering whether there's any code in PETSc that I can either use directly or lift out and adapt for my purposes (which are described in footnote [1]).<br>
<br>
What I need to do is a number of different 2D array ("multi-vector", "dense matrix") repacking operations, in order to make the data available to different operations (spherical harmonic transforms, dense linear algebra, sparse linear algebra, etc.).<br>
<br>
There's a number of repacking operations I'm in need of:<br>
<br>
- From each process having a given contiguous set of rows of a 2D array, to an element-cyclic distribution where one distributes over both rows and columns (the distribution of Elemental [2])<br>
<br>
- Arbitrary redistribution of rows between processes (but each row is kept in a process), potentially with overlaps.<br>
<br>
- More structured redistribution of rows between processes where indices can be easily computed on the fly (but still a pattern quite specific to my application).<br>
<br>
While I found PETSc routines for arbitrary redistribution of 1D arrays, I couldn't find anything for 2D arrays. Any pointers into documentation or papers etc. explaining these aspects of PETSc is very welcome.<br></blockquote>
<div><br></div><div>It sounds like you just want help with indexing. VecScatter can move data of any dimensionality, you just have to know how to index it. (You could ask for something more specialized if you were using the topology routines in MPI and wanted control over how messages are routed.)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
What I'd *really* like (and what I'll do if I end up doing it myself) is something more generic: A small, focused library which, given two descriptors of how an N-dimensional array is distributed, does the redistribution from one to the other. What I'd imagine is that you for each axis of the array pick an axis distribution ("chunked", "(block-)cyclic", "provided by indices", "non-distributed"), and an MPI datatype. (The array would have to be distributed on a process grid of the same dimension as the array).<br>
<br>
Then, given two N-dimensions descriptors which mixes structured and unstructured information, the library can make and execute a redistribution plan.<br>
<br>
Perhaps too ambitious, which is why I'm hoping that somebody can point me to at least something existing. I can imagine doing most of it from scratch, but reusing PETSc for the unstructured axis-distribution type.<br>
<br>
Comments of any kind welcome.<br>
<br>
Dag<br>
<br>
[1] What I'm doing is solving a linear system by CG where the LHS matrix can only be written as a (rather large) symbolic equation, containing lots of spherical harmonic transforms, Fourier transform, both dense and sparse linear algebra, and so on.<br>
<br>
[2] <a href="http://code.google.com/p/elemental/" target="_blank">http://code.google.com/p/<u></u>elemental/</a><br>
</blockquote></div><br></div>