[mpich-discuss] MPI_Allreduce increases the performance of MPI_Alltoallv?

[Rusty Lusk]

Could you send us the Jumpshot images you refer to?

Thanks,
Rusty

On Tuesday,Jul 3, 2012, at 1:39 PM, Li, Lihua (UMSL-Student) wrote:

> Dear MPICH users,
> 
> Do any one happen to have such an experience, when MPI_Allreduce "seems to" make MPI_Alltoallv faster? I am currently stuck on this performance issue and could not figure out why. The project I am working on is a parallel version of conjugate gradient solver, which would rely on MPI calls to update the vectors(MPI_Alltoallv) and scalars (MPI_Allreduce) values. In the original and most simple implementation, MPI_Alltoallv calls interleaves MPI_Allreduce calls. We modified the algorithm a little bit to put the data transmitted in MPI_Allreduce into MPI_Alltoallv. So the new, modified algorithm would have the same number of MPI_Alltoallv calls, same pattern of data transmission, slightly larger trunk of data transmitted in MPI_Alltoallv (~100bytes more, original transmission 100Mbytes+) and no MPI_Allreduce calls. The result is a bit surprising to us. Instead of a performance gain, the modified algorithm shows a slight performance loss compared to the unmodified algorithm after repeated experiments. Further timing of different parts of the code shows the performance discrepancy lies in MPI_Alltoallv and MPI_Allreduce calls.
> 
> We looked into the Jumpshot images trying to find out the reason. The two jumpshots looks rather similar to each other, since the transmission pattern is the same. However, the Jumpshots of the unmodified algorithm looks more "regular": the time spent on MPI_Alltoallv is about the same for every process, while the MPI_Allreduce operations takes very little time. The Jumpshot image for the modified version has greater variation of MPI_Alltoallv time among processes, while there is no MPI_Allreduce operations. It gives me a weird feeling that MPI_Allreduce operations are "regulating" the behavior of MPI_Alltoallv.
> 
> Anyone got hint of what's going on? 
> 
> Regards,
> Tom Li
> 
> 
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20120703/0095606c/attachment.html>