[MPICH2-dev] MPID_PG_ForwardPGInfo

[William Gropp]

At 06:59 PM 12/21/2005, Charles J Archer wrote:

>Hello
>
>I've recently imported a new release of mpich (1.0.3), and I noticed that 
>the ch3 device defines a new function:
>int MPID_PG_ForwardPGInfo( MPID_Comm *peer_ptr, MPID_Comm *comm_ptr,
>                            int nPGids, int gpids[],
>                            int root )
>
>This function is used by the intercomm_create function.
>This seems to fill out new records in the VC record fields.
>
>On BG/L, we keep our VC records intentionally very small so that we scale 
>to thousands of nodes.
>I would like to avoid adding the extra structures required to implement 
>this function on BG/L.
>
>Essentially, what this amounts to is that changes to the CH3 device are 
>forcing changes to the MPI functions, breaking the ADI abstraction.
>Consequently, all other devices have to implement any new function that 
>has been added to CH3.
>
>To work around this, I simply defined the function with an empty 
>implementation.  I'm not sure if this is sufficient (it appears to pass 
>the icXXX) tests on BG/L.
>Please advise on how I should handle this in the BG/L device.

Charles,
Sorry about this.  This was a quick fix to handle problems in correctly 
managing dynamic processes and in fact there are new tests in the 
test/mpi/spawn that require something like this to pass.  We will need an 
additional ADI function for this case, but only when the dynamic process 
routines are supported.  Also, as you note, we shouldn't make the VC 
structures visible (and in fact, some of these new routines were intended 
to reduce rather than increase the use of VC internals).  We need to fix 
both of these problems.

I'd like to get the interface for this right for the next release.  That 
might mean moving more of the intercomm support, at least optionally, into 
the device, in order to handle the general case.  One possibility is to 
special case the situation where dynamic processes are not supported; this 
can be essentially the same as the pre 1.0.3 version.  Another is to wrap 
up the operations needed for intecomm create in the general case and allow 
the device to provide those functions; this would allow the device complete 
freedom to handle the scalability issue without dictating a particular 
approach or data structure.  Another issue to consider is optimizing for 
the single MPI_COMM_WORLD case but provide hooks that allow the 
implementation to handle the more general case as required.  I'm interested 
in your thoughts on what is appropriate for both BG/L and 
BG/<fill-in-the-blank> :) .

I'm sending this out now to give us all some time to mull it over; Argonne 
closes today and I'm already on vacation.  We're back January 3rd.

Bill




>Thanks!
>
>-C
>
>Charles J Archer
>eServer Consolidation Development
>BlueGene/L MPI
>eServer Development
>Dept G9D, Bldg 030-2,  IBM Rochester MN 55901
>archerc at us.ibm.com   (507) 253-0346     TL 8-553-0346   Fax (507) 253-2870

William Gropp
http://www.mcs.anl.gov/~gropp