[petsc-dev] IMP prototype

[Victor Eijkhout]

On Dec 31, 2013, at 2:37 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> How would you propose to handle

Look, what I have now is basically like a re-implementation of a VecScatter between two distributed vectors. It's only partly about coming up with cute new algorithms under the hood: my foremost claim is about the integrative API.

> irregular communication patterns

Sparse matrices? Basically a VecScatter.

> such as
> you would get with a small number of dense rows in a sparse matrix, or
> the row/column communicators for dense linear algebra?  How are
> reductions expressed and implemented?

Are the two questions related? Reductions: my theory paper describes it. I have no implementation yet.

Small number of dense rows: are you implying that a matrix-vector product with such a matrix should be a combination of regular sparse VecScatter and a reduction? Interesting thought. As a refinement many miles down the line I could think about it.

> I think that the setup costs are actually critical.  They are not always
> amortized in either dynamic execution models or methods with frequent
> adaptivity, particle transport, self-contact, etc.  The MPI model may
> already have two-sided knowledge about the new communication pattern or
> know a good way to use collectives.  It's more work for the programmer
> to specify both sides of the communication, but dynamic exchange is
> significantly more expensive.

I'm not really sure what you're arguing here, but I'll offer the following.
1. any setup algorithm you have in Petsc I can straightaway copy to my software
2. adaptivity: rather than doing a full setup you could do an update of an existing one. Since adaptive refinement or load balancing or such means a change of one of the distributions involved, this is a formal transformation just like the alpha-beta transform. Therefore the system can calculate the effect of it. I could implement it by composition (as in: execute the one after the other) of two transformations. And then I could think about a transformation of transformations that collapses them into one, which would be a local update of the communication setup. This is also several miles down the line.

Thanks for your thoughts. If I could get someone to fund this stuff me and a couple of students/post-docs could have no ends of fun turning this into useful software.

Victor.
-- 
Victor Eijkhout, 512 471 5809 (w)
Texas Advanced Computing Center, 
The University of Texas at Austin