[petsc-dev] Generality of VecScatter

[Jed Brown]

I had a useful conversation with the MPICH guys today about the
unstructured communication primitives that Mark, Matt, Barry, and I have
been discussing. It's looking like we can do a very thin communication
library that has the nice semantics I was looking for, namely that the
local-to-global is stored only as indices in the local space. To implement
the communication primitives, the owner of a global node never needs to
know how many or which processes have that point in their local space.
There is an MPI-3 feature that we can use to elegantly avoid two-way
knowledge for pointwise gather/scatter. (We can always implement the same
interface using vanilla MPI-1, but the MPI-2/3 implementation would have
less overhead.)

If I/we write this layer, VecScatter (with arbitrary data types, etc) could
be implemented very easily on top of it. I'm in no rush to do this, but I
could imagine getting there eventually. The VecScatter interface allows
some things that might not be important. For example, any process can
specify an edge in the communication graph, even if it does not own the
source or the destination vertex. Also, both source and destination
vertices can have degree higher than one. We can even put a link between
the same points twice (effectively multiplying that contribution by 2 when
using ADD_VALUES). I don't know if this ever makes semantic sense, but the
interface allows it.

I wonder if we can express all useful communication with a more restrictive
interface similar to a local-to-global map (but mapping between any
vectors) where one side (typically called "local", but doesn't need to
actually be a local Vec) has at most one edge.

What weird things do people use VecScatter for that might break with this
more restrictive model?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20111123/f5fec4b2/attachment.html>