[MPICH] Implementation of MPI_Alltoallw

[Gerard Gorman]

The classic idea is that you let communication overlap computation but I 
think you still need to think a bit more about Rajeev's point as most 
people don't bring this up - when everyrank needs to communicate with 
everyrank you can post isends&irecvs and let the system deal with it. I 
remember some time ago it was popular to implement algorithms on the 
application side to use blocking sends to try to optimize throughput. 
Other than the fact that the benefits are not generally portable, you 
don't know everything a priori. For example there may be congestion in 
some of the switches due to other applications on the cluster and so it 
might be better to complete other communications while the congestion is 
reduced or your message gets through the bottle neck. Basically, outside 
your bit of code anything could be happening. I would guess that 
research into this is really on the OS/switch level.

Cheers
g

PS. the cost of initialising mpi data types is very small

Sudarshan Raghunathan wrote:

>Rajeev,
>
>Thank you for your response. I was under the impression that
>non-blocking sends and receives are most useful when one is
>overlapping communication and computation steps. This does not seem to
>be the case in the MPICH implementation of MPI_Alltoallw since there
>is no computation per se - all the information is know a priori.
>
>However, I would guess that if one were using custom data types (like
>vectors or sub arrays), then the creation of the data types could
>perhaps be overlapped with the communication steps (of course, one
>could then not call MPI_Alltoallw directly, but just post the isends
>and irecvs). I think that gives me something to experiment :-)
>
>Regards,
>Sudarshan
>
>
>  
>