[petsc-dev] MPIX_Iallreduce()

[Jed Brown]

On Tue, Mar 20, 2012 at 00:06, Jed Brown <jedbrown at mcs.anl.gov> wrote:

> On Sun, Mar 18, 2012 at 11:55, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>  Add a glance adding all these new complications to PETSc to chase an
>> impossible overlap of communication and computation sounds fine :-)
>
>
> /Q has a dedicated thread to drive asynchronous comm. I've added this, the
> call to PetscCommSplitReductionBegin() is entirely optional (does not alter
> program semantics), but will allow asynchronous progress to be made.
>
> On conventional systems, there are two choices for driving asynchronous
> progress:
>
> 1. Set the environment variable MPICH_ASYNC_PROGRESS=1. "Setting that
> environment variable will cause a cheesy form of background progress
> wherein the library will spawn an additional background thread per MPI
> process.  You'll have to play around with things, but I'd recommend cutting
> your number of processes per node in half to avoid nemesis oversubscription
> badness." -- Dave Goodell
>
> 2. Make any nontrivial calls into the MPI stack. This could specifically
> mean polling a request, but it could also just be your usual communication.
> I suspect that a standard MatMult() and PCApply() will be enough to drive a
> significant amount of progress on the reduction.
>
> http://petsc.cs.iit.edu/petsc/petsc-dev/rev/d2d98894cb5c
>

And here's some preliminary test output on 32 processes of cg.mcs. I ran a
handful of times with src/vec/vec/examples/tests/ex42.c and these results
seem reproducible.

VecScatterBegin      300 1.0 8.5473e-04 1.5 0.00e+00 0.0 9.6e+03 4.0e+01
0.0e+00  2  0 98 99  0   2  0 98 99  0     0
VecScatterEnd        300 1.0 1.1136e-02 7.6 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 18  0  0  0  0  18  0  0  0  0     0
VecReduceArith       100 1.0 1.4210e-04 1.5 9.00e+02 1.0 0.0e+00 0.0e+00
0.0e+00  0100  0  0  0   0100  0  0  0   203
VecReduceComm        100 1.0 2.1405e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00
1.0e+02 60  0  0  0 89  60  0  0  0 90     0

VecScatterBegin      300 1.0 5.9962e-04 1.9 0.00e+00 0.0 9.6e+03 4.0e+01
0.0e+00  5  0 98 99  0   5  0 98 99  0     0
VecScatterEnd        300 1.0 1.6668e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 12  0  0  0  0  12  0  0  0  0     0
VecReduceArith       100 1.0 1.1611e-04 2.5 9.00e+02 1.0 0.0e+00 0.0e+00
0.0e+00  1100  0  0  0   1100  0  0  0   248
VecReduceBegin       100 1.0 5.3430e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  4  0  0  0  0   4  0  0  0  0     0
VecReduceEnd         100 1.0 2.4858e-03 2.6 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 21  0  0  0  0  21  0  0  0  0     0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120320/7e612409/attachment.html>