[mpich-discuss] communicator creation/deletion semantics and performance

[Edgar Solomonik]

Hello,

For my application, I need to maintain or dynamically create a large number
of communicators.  My current solution has been to initialize a large
number of communicators at start-up and make dynamic decisions on which to
use later.  I have ran into MPI errors due to creating too many
communicators on some occasions, but have so far been able to resolve this
by limiting the set.

However, I am now interested in employing an even larger set of
communicators, that is harder to generate completely.  So, I would like to
move to an approach which dynamically creates and frees communicators on
demand.  I am concerned about two issues:

1. Is there an overhead to MPI_Comm_split and MPI_Comm_free, for instance
do they need to perform inter-process communication?
2. Does the limit on the number of communicators bound the number of
communicators ever created or the number of live (non-freed) communicators?

My specific use-case is merging sets of communicators in dynamic ways.
 e.g. on BG/Q I form up 6 communicators for each dimension (+1 for
intra-node) and then make dynamic mapping decisions which select unions of
the communicators to map to.  So, I either need to construct a fairly
complicated tree data-structure to keep up with all possible unions of
communicators or I can simply create the unions on demand and free them
once I am done using them after a given iteration.  So, far I had used only
contiguous unions of communicators, which is a smaller set and easier to
keep track of in a flat data-structure, but I want even more generality now.

Thanks,
Edgar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20121030/769753bc/attachment.html>