[petsc-dev] MPIX_Iallreduce()

[Matthew Knepley]

On Sun, Mar 18, 2012 at 11:06 AM, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> We currently have Vec{Norm,Dot,TDot,MDot,MTDot}{Begin,End}() allowing the
> reductions to be aggregated, but the reduction itself is only triggered
> lazily and always uses MPI_Allreduce(). MPICH2 has implemented
> MPI_Iallreduce (named MPIX_Iallreduce() until MPI-3 is finalized). I
> suggest adding PetscCommSplitReductionBegin(MPI_Comm) which would start any
> currently queued split reductions. Then, a latency-tolerant algorithm might
> be written
>
> VecNormBegin(..,&nrm);
> VecDotBegin(...,&dot);
> PetscCommSplitReductionBegin(comm);
> MatMult(...); // Or residual evaluation, etc.
> VecNormEnd(...,&nrm);
> VecDotEnd(...,&dot);
>
>
> PetscCommSplitReductionBegin() would start the split reduction, leaving
> behind an MPI_Request that would be waited on by the first XXEnd().
>
> If MPIX_Iallreduce() is not available, the current semantics would be
> used. If you don't call PetscCommSplitReductionBegin(), the present
> semantics would also be used.
>
> Is this a good API?
>

This sounds fine to me. I think the acid test is to rewrite Bill's new CG
in it.

I'd also like to propose a design change in which the PetscSplitReduction
> is placed in _p_Vec when it is gotten out of the MPI_Comm. The MPI_Comm
> would continue to hold a reference until PetscCommSplitReductionBegin() at
> which point it drops the reference and the only remaining references are
> held by the Vecs participating in the split reduction. That would enable a
> new split reduction to be done by other objects using the same
> communicator, before the results of the last split reduction have been
> collected by all participants.
>

I do not understand this. What call creates the PetscSplitReduction struct?
Is it the Begins? So after Begin, it is present in
the Vec and the Comm. Then when you call PetscCommSplitReductionBegin, it
leaves the comm, and resides only in the
Vec along with a Request. Then the End destroys it? So it lives in the Comm
for aggregation purposes? That sounds fine to
me.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120318/22318f4a/attachment.html>