Thank you for the responses.<div><br></div><div>It sounds like dynamic communicator creation/deletion is good enough for my purposes. I looked into the papers and I don't think the advantages of create_group are important in my scenario (global synchronization is not an issue). I will probably do this with comm_split, and look into the MPI-3 functionality if split becomes a bottleneck.</div>
<div><br></div><div>Thanks again,</div><div>Edgar</div><div><br><div class="gmail_quote">On Tue, Oct 30, 2012 at 11:14 AM, Jim Dinan <span dir="ltr"><<a href="mailto:dinan@mcs.anl.gov" target="_blank">dinan@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The paper that Jeff cited presents an implementation of MPI_Comm_create_group that is built on top of MPI-2 and uses recursive intercommunicator merging:<br>
<br>
<a href="https://www.mcs.anl.gov/publications/paper_detail.php?id=1695" target="_blank">https://www.mcs.anl.gov/<u></u>publications/paper_detail.php?<u></u>id=1695</a><br>
<br>
At this year's EuroMPI, we presented a paper on the native MPI-3 implementation of MPI_Comm_create_group and compared its cost with MPI_Comm_create and the technique presented in the above paper at EuroMPI '11. The general takeaway is that MPI_Comm_create_group should always be the cheaper:<br>
<br>
<a href="https://www.mcs.anl.gov/publications/paper_detail.php?id=2061" target="_blank">https://www.mcs.anl.gov/<u></u>publications/paper_detail.php?<u></u>id=2061</a><span class="HOEnZb"><font color="#888888"><br>
<br>
~Jim.</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On 10/30/12 12:48 PM, Jeff Hammond wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If you want to dynamically generate communicators on a group without<br>
using it being collective on the parent communication (which might be<br>
MPI_COMM_WORLD in the worst case), see<br>
<a href="http://www.mcs.anl.gov/publications/paper_detail.php?id=1695" target="_blank">http://www.mcs.anl.gov/<u></u>publications/paper_detail.php?<u></u>id=1695</a>. I<br>
suspect this will be useful for doing the dynamic unions you describe<br>
below. This operation is part of MPI-3 (see MPI_Comm_create_group)<br>
and was implemented in MPICH2 a while ago. I don't know the optimized<br>
(i.e. internal) implementation works on BGQ but I suspect it can be<br>
done relatively easily in the event that the on-top-of-MPI<br>
implementation isn't fast enough (it is quite fast - faster than<br>
MPI_Comm_create - for small groups, at least on BGP (see paper for<br>
details)).<br>
<br>
I believe that MPI_Comm_split does communication, potentially<br>
expensive communication (allgather? or maybe that was the optimized<br>
version...). At some point, I recall Bill Gropp and coworkers doing<br>
some work to optimize it because of a problem we observed at scale on<br>
BGP. I don't know the status of that and whether or not it is part of<br>
MPICH2 yet. As for MPI_Comm_free, I suspect it is quite cheap and<br>
does at most a barrier, but I am speculating.<br>
<br>
I can hack an optimized version of MPI_Comm_create_group into BGQ-MPI<br>
using PAMI if IBM doesn't do it first.<br>
PAMI_Geometry_create_<u></u>endpointlist is collective only over the output<br>
geometry, which matches the semantics of MPI_Comm_create_group.<br>
<br>
Is this at all helpful?<br>
<br>
Jeff<br>
<br>
On Tue, Oct 30, 2012 at 11:33 AM, Edgar Solomonik<br>
<<a href="mailto:solomon@eecs.berkeley.edu" target="_blank">solomon@eecs.berkeley.edu</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
For my application, I need to maintain or dynamically create a large number<br>
of communicators. My current solution has been to initialize a large number<br>
of communicators at start-up and make dynamic decisions on which to use<br>
later. I have ran into MPI errors due to creating too many communicators on<br>
some occasions, but have so far been able to resolve this by limiting the<br>
set.<br>
<br>
However, I am now interested in employing an even larger set of<br>
communicators, that is harder to generate completely. So, I would like to<br>
move to an approach which dynamically creates and frees communicators on<br>
demand. I am concerned about two issues:<br>
<br>
1. Is there an overhead to MPI_Comm_split and MPI_Comm_free, for instance do<br>
they need to perform inter-process communication?<br>
2. Does the limit on the number of communicators bound the number of<br>
communicators ever created or the number of live (non-freed) communicators?<br>
<br>
My specific use-case is merging sets of communicators in dynamic ways. e.g.<br>
on BG/Q I form up 6 communicators for each dimension (+1 for intra-node) and<br>
then make dynamic mapping decisions which select unions of the communicators<br>
to map to. So, I either need to construct a fairly complicated tree<br>
data-structure to keep up with all possible unions of communicators or I can<br>
simply create the unions on demand and free them once I am done using them<br>
after a given iteration. So, far I had used only contiguous unions of<br>
communicators, which is a smaller set and easier to keep track of in a flat<br>
data-structure, but I want even more generality now.<br>
<br>
Thanks,<br>
Edgar<br>
<br>
______________________________<u></u>_________________<br>
mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/<u></u>mailman/listinfo/mpich-discuss</a><br>
<br>
</blockquote>
<br>
<br>
<br>
</blockquote>
______________________________<u></u>_________________<br>
mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/<u></u>mailman/listinfo/mpich-discuss</a><br>
</div></div></blockquote></div><br></div>