[mpich-discuss] MPI_Allgatherv complexity

[Justin Luitjens]

Hi,

I also contacted Kim McMahon of Cray about this.  He told me this:


===========================================
Hi Justin,

Hmm. This performance looks pretty bad.  This reminds me of some
MPI_Allgather performance results I gathered several months ago.
We found that the MPICH2 MPI_Allgather algorithm was not scaling
on Cray's XT systems.  At that point, I re-wrote the MPI_Allgather
algorithm for XT.  Given your data, it looks like I might need to
do the same for MPI_Allgatherv.  As of our current release, we are
still using the MPICH2 algorithm for MPI_Allgatherv.

Is there any chance you can substitute an MPI_Allgather for your
MPI_Allgatherv calls?

I will look deeper into the Allgatherv performance in the next
few weeks.  I'll let you know if I have something new for you to
try.

Thanks for bringing this to our attention.

-Kim
=================================================


I decided to change my test program to use all gather and performed the same
tests.  The results are attached.  As you can see the performance is much
better.  I will try and figure out if cray has the new algorithm
implemented.  This might also be a problem with XT.

Justin
On Mon, Nov 2, 2009 at 5:37 PM, Pavan Balaji <balaji at mcs.anl.gov> wrote:

>
> On 11/02/2009 06:27 PM, Justin Luitjens wrote:
> > Actually I just realized I was wrong with what I said.  Our code has a
> > lot of variation but the allgather test that I wrote and timed in the
> > graph I attached will only have a single element of variation between
> > each process.  Is there an environment variable I can set to force
> > either the ring algorithm or the recursive doubling algorithm to compare
> > the times?
>
> No, there's no such provisioning (though we should probably add it).
>
> This does look like a performance issue. Can you create a ticket here:
> https://trac.mcs.anl.gov/projects/mpich2/newticket
>
> Thanks,
>
>  -- Pavan
>
> --
> Pavan Balaji
> http://www.mcs.anl.gov/~balaji <http://www.mcs.anl.gov/%7Ebalaji>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20091102/56ea2509/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: allgather.png
Type: image/png
Size: 18692 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20091102/56ea2509/attachment-0001.png>