[petsc-dev] MPIX_Iallreduce()

[Jed Brown]

We currently have Vec{Norm,Dot,TDot,MDot,MTDot}{Begin,End}() allowing the
reductions to be aggregated, but the reduction itself is only triggered
lazily and always uses MPI_Allreduce(). MPICH2 has implemented
MPI_Iallreduce (named MPIX_Iallreduce() until MPI-3 is finalized). I
suggest adding PetscCommSplitReductionBegin(MPI_Comm) which would start any
currently queued split reductions. Then, a latency-tolerant algorithm might
be written

VecNormBegin(..,&nrm);
VecDotBegin(...,&dot);
PetscCommSplitReductionBegin(comm);
MatMult(...); // Or residual evaluation, etc.
VecNormEnd(...,&nrm);
VecDotEnd(...,&dot);


PetscCommSplitReductionBegin() would start the split reduction, leaving
behind an MPI_Request that would be waited on by the first XXEnd().

If MPIX_Iallreduce() is not available, the current semantics would be used.
If you don't call PetscCommSplitReductionBegin(), the present semantics
would also be used.

Is this a good API?


I'd also like to propose a design change in which the PetscSplitReduction
is placed in _p_Vec when it is gotten out of the MPI_Comm. The MPI_Comm
would continue to hold a reference until PetscCommSplitReductionBegin() at
which point it drops the reference and the only remaining references are
held by the Vecs participating in the split reduction. That would enable a
new split reduction to be done by other objects using the same
communicator, before the results of the last split reduction have been
collected by all participants.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120318/e9baad99/attachment.html>