[mpich-discuss] MPI_Allgatherv complexity

Justin Luitjens luitjens at cs.utah.edu
Mon Nov 2 20:58:05 CST 2009


I also contacted Kim McMahon of Cray about this.  He told me this:

Hi Justin,

Hmm. This performance looks pretty bad.  This reminds me of some
MPI_Allgather performance results I gathered several months ago.
We found that the MPICH2 MPI_Allgather algorithm was not scaling
on Cray's XT systems.  At that point, I re-wrote the MPI_Allgather
algorithm for XT.  Given your data, it looks like I might need to
do the same for MPI_Allgatherv.  As of our current release, we are
still using the MPICH2 algorithm for MPI_Allgatherv.

Is there any chance you can substitute an MPI_Allgather for your
MPI_Allgatherv calls?

I will look deeper into the Allgatherv performance in the next
few weeks.  I'll let you know if I have something new for you to

Thanks for bringing this to our attention.


I decided to change my test program to use all gather and performed the same
tests.  The results are attached.  As you can see the performance is much
better.  I will try and figure out if cray has the new algorithm
implemented.  This might also be a problem with XT.

On Mon, Nov 2, 2009 at 5:37 PM, Pavan Balaji <balaji at mcs.anl.gov> wrote:

> On 11/02/2009 06:27 PM, Justin Luitjens wrote:
> > Actually I just realized I was wrong with what I said.  Our code has a
> > lot of variation but the allgather test that I wrote and timed in the
> > graph I attached will only have a single element of variation between
> > each process.  Is there an environment variable I can set to force
> > either the ring algorithm or the recursive doubling algorithm to compare
> > the times?
> No, there's no such provisioning (though we should probably add it).
> This does look like a performance issue. Can you create a ticket here:
> https://trac.mcs.anl.gov/projects/mpich2/newticket
> Thanks,
>  -- Pavan
> --
> Pavan Balaji
> http://www.mcs.anl.gov/~balaji <http://www.mcs.anl.gov/%7Ebalaji>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20091102/56ea2509/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: allgather.png
Type: image/png
Size: 18692 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20091102/56ea2509/attachment-0001.png>

More information about the mpich-discuss mailing list