[MPICH2-dev] ordering of reduction operations

[Larry Stewart]

I have heard somewhere, but I can't remember where, that it is important
for an implementation of reductions to have a consistent ordering of
the operations <<when it matters>>.

For example, one could compute MIN or MAX in any order, but
for SUM with floating point arguments it might make a real (small joke)
difference to do the adds in a different order.

My understanding is that the MPI spec is silent on this issue, but that
folks with applications tend to get testy when their codes give different
answers from run to run.

It probably doesn't matter <what> order is chosen, but should the
implementation be required to use the <same> order from run to run?

In a flat interconnect like IB with a single switch, there is no obvious
preferred order.  In a system where some ranks run on the same SMP
while other ranks are across the network, it might be faster to combine
locally before going off-machine.  For multi-hop networks like BG/L, one
can get a lot more performance by scheduling the collective's use of links.

More questions:  If the ordering needs to be consistent, can the
choice depend on, for example, the length of the reduction?  (One might
choose different algorithms for small and for large, with different 
preferred
orderings.)

Whoever told me this also mentioned that it was a source of constant
irritation that different MPI implementations had different orderings,
leading to different answers when the application was linked with a
different package.

Q: Is this a real issue or have I been in the sun too long?

-Larry