Actually I just realized I was wrong with what I said. Our code has a lot of variation but the allgather test that I wrote and timed in the graph I attached will only have a single element of variation between each process. Is there an environment variable I can set to force either the ring algorithm or the recursive doubling algorithm to compare the times? <br>
<br>Justin<br><br><div class="gmail_quote">On Mon, Nov 2, 2009 at 5:17 PM, Justin Luitjens <span dir="ltr"><<a href="mailto:luitjens@gmail.com">luitjens@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
There is likely a decent amount of variance in the distribution.<div><div></div><div class="h5"><br><br><div class="gmail_quote">On Mon, Nov 2, 2009 at 5:15 PM, Pavan Balaji <span dir="ltr"><<a href="mailto:balaji@mcs.anl.gov" target="_blank">balaji@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><br>
On 11/02/2009 06:11 PM, Justin Luitjens wrote:<br>
> Here is a graph of the time for an allgather as a function of message<br>
> size. We would like to scale up to 98K processors but as you can see<br>
> the time for all gather dramatically increases at 49K processors. At<br>
> 98K the time increases by around a factor of 4 for the few datapoints<br>
> that I have tested (they are not in the graph).<br>
<br>
</div>What's the message size pattern like? Is everyone using about the same<br>
size messages, or is there a lot of skew? For the second case, we added<br>
a new algorithm to fix this performance issue, though I'm not sure Cray<br>
picked it up yet.<br>
<br>
-- Pavan<br>
<font color="#888888"><br>
--<br>
Pavan Balaji<br>
<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
</font><div><div></div><div>_______________________________________________<br>
mpich-discuss mailing list<br>
<a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>