[mpich-discuss] Best way to do "Allgather to subcommunicator"?

[Dave Goodell]

Point-to-point (#1) is probably your best bet if you need memory scalability.  I suspect that #2 would be faster under many existing implementations, but even faster algorithms should exist for a direct solution.  I'm not sure where #3 would fall relative to the first two.

You might also be able to express this as a neighborhood collective operation with a graph topology (haven't thought hard about this yet), although those are obviously unoptimized right now.  We could look at this as a motivating example for internal MPI optimization techniques.

Tom's DIY library may also implement this pattern in some way.  And if it doesn't, you could see if it's a pattern he feels like supporting: http://www.mcs.anl.gov/~tpeterka/software.html

-Dave

On Sep 10, 2012, at 8:24 AM CDT, Jed Brown wrote:

> Given an original communicator of size P and S subcomms each of size P/S, I effectively want to do an "Allgather" with the result distributed over each of the S subcomms. Some options include
> 
> 1. point-to-point
> 2. Allgather, then each process keeps the part that is most relevant to them (limited by memory)
> 3. Creating subcomms for each stratum in the S subcomms, Gather to leader, then Bcast along strata
> 
> Are there better ways to do this? The simpler "Gather to subcomm" operation is also useful.
> 
> Background: this redundant redistribution is one approach to relieving coarse grid bottlenecks in multigrid. Making it faster would definitely have a tangible impact on solver performance.
> _______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss