[mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Tue Jun 28 17:06:45 CDT 2011

Hi,
 The code in smpd_ipmi.c implements the PMI library (PMI v1 APIs). This part is currently integrated into the MPI library (mpi.lib). You should look at *spawn() in this code and look into the SMPD process manager code (src/pm/smpd/*) to see how the PMI library interacts with the process manager. Your process manager should implement parts in the SMPD process manager code (src/pm/smpd/*) corresponding to spawn functionality to avoid recompilation (previously compiled with MPICH2 libraries) of your MPI programs (and avoid building the MPI library).

(PS: Meanwhile, I would note that Hydra has a much better design than SMPD. I would recommend looking at Hydra to understand the required design and implement the various components. The hydra architecture is explained at http://wiki.mcs.anl.gov/mpich2/index.php/Hydra_Process_Management_Framework)
Regards,
Jayesh

----- Original Message -----
From: "Eric Hui" <Eric.Hui at ggy.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Cc: mpich-discuss at mcs.anl.gov
Sent: Tuesday, June 28, 2011 3:15:58 PM
Subject: RE: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Hi Jayesh,

Right now my service application WinResource.exe already has all the information provided by mpiexec.  So if I add the code similar to smpd_ipmi.c into WinResource.exe and make it compatible with the current PMI library that's compiled into the MPI program (mpi.lib), then I don't need to recompile the mpi.lib and can launch MPIApp.exe without mpiexec, is my understanding correct?

I just want to make sure I am getting the correct big picture before going into the implementation.

Thanks for all your quick responses!
Eric

-----Original Message-----
From: Jayesh Krishna [mailto:jayesh at mcs.anl.gov] 
Sent: Tuesday, June 28, 2011 4:05 PM
To: Eric Hui
Cc: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Hi,
 The role of mpiexec is to launch MPI programs. Therefore mpiexec provides options that allow users for example to specify the nodes to launch the MPI program, the number of MPI processes to launch etc (If you replace mpiexec you need to have a mechanism to provide this information).
 The support for dynamic processes (MPI_Comm_spawn() etc) should be handled by the MPI library with the help of possibly a PMI library. mpiexec does not come into picture here.

(PS: However note that if you replace the PMI library, all the MPI programs would required to be recompiled with the new pmi (mpi) library)

Regards,
Jayesh

----- Original Message -----
From: "Eric Hui" <Eric.Hui at ggy.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Cc: mpich-discuss at mcs.anl.gov
Sent: Tuesday, June 28, 2011 2:47:20 PM
Subject: RE: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Hi Jayesh,

Thanks a lot for the information.

Currently I am dealing with 3 executables:  smpd.exe, mpiexec.exe and my Windows Application (let's call it MPIApp.exe).

Can you confirm if my understanding is correct?

Supposed my Windows Service app used for controlling the resource is called WinResource.exe.  Currently we use WinResource.exe to launch MPIApp.exe.  If we implement our own wire protocol, then WinResource.exe can replace the role of both SMPD.exe and mpiexec.exe and simply launch the MPIApp.exe to call MPI_Comm_Spawn or a combination of MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect?  Or do I still need to use mpiexec.exe?  This is the part that I am not sure about.

Thanks!
Eric

-----Original Message-----
From: Jayesh Krishna [mailto:jayesh at mcs.anl.gov] 
Sent: Tuesday, June 28, 2011 3:32 PM
To: Eric Hui
Cc: mpich-discuss at mcs.anl.gov
Subject: Re: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Hi,
 The PMI wire protocol is used for communicating between the PMI library (compiled into the MPI program) and the process manager (like SMPD).
 You can replace the PMI library (look into https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk/src/pmi/smpd/smpd_ipmi.c) to change the wire protocol to be compatible with a custom wire protocol that is supported by your process manager OR implement the custom wire protocol used by the PMI library (like SMPD) in your process manager.
 Alternately you can replace both the PMI library and the process manager to implement your own wire protocol. If this is the route you take I would recommend looking into the current PMI2 implementation (PMI library - https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk/src/pmi/pmi2/* & Hydra process manager - https://svn.mcs.anl.gov/repos/mpi/mpich2/trunk/src/pm/hydra/pm/pmiserv/*) and developer documentation related to PMI2 (http://wiki.mcs.anl.gov/mpich2/index.php/PMI_v2_Wire_Protocol).
 PMI (v1) only defined the interface to the process manager (http://www.mpi-forum.org/docs/mpi-20-html/node89.htm#Node89) and multiple vendors have supported this interface. PMI2 also defines the wire protocol (and also has enhancements to PMI v1) allowing vendors to choose to implement just the process manager compatible with PMI wire protocol as discussed above. 
 Hope this helps.

Regards,
Jayesh

----- Original Message -----
From: "Eric Hui" <Eric.Hui at ggy.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>, mpich-discuss at mcs.anl.gov
Sent: Tuesday, June 28, 2011 2:13:53 PM
Subject: RE: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

Hi Jayesh,

We are currently using WinSock, so there is no PMI protocol code.  That's why I am starting this investigation.

Suppose that we can add support for the wire protocol in our Windows Service application, then do we still need to use mpiexec / smpd?

What I am trying to figure out is what the mpiexec does when I call MPI_Comm_Spawn or a combination of MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect?

Finally, is PMI specific to the MPICH2 implementation?  I was reading the MPI 2.2 spec and doesn't seem to find any reference to it.

Eric

-----Original Message-----
From: Jayesh Krishna [mailto:jayesh at mcs.anl.gov] 
Sent: Tuesday, June 28, 2011 3:05 PM
To: mpich-discuss at mcs.anl.gov
Cc: Eric Hui
Subject: Re: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

 Does your custom process manager (Windows Service) support the PMI wire protocol (protocol used by the PMI library to interact with the process manager)? PMI2 (The newer version of PMI) defines/standardizes the wire protocol and is implemented in the new process manager Hydra. Unfortunately I don't have enough resources to port Hydra to Windows (SMPD only supports PMI-1 and has its own custom wire protocol that you will have to figure out by looking into the code and the debug logs).

Regards,
Jayesh

----- Original Message -----
From: "Eric Hui" <Eric.Hui at ggy.com>
To: mpich-discuss at mcs.anl.gov
Sent: Thursday, June 23, 2011 10:19:26 AM
Subject: [mpich-discuss] MPI_Comm_Spawn and MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect without using mpiexec and smpd

I just started to investigate the possibility of using MPI in our application on the Windows platform. 

Here is what I am trying to accomplish: 

I am trying to use both of the following techniques: 
1. MPI_Comm_Spawn 
2. Client/server model using MPI_Open_port/MPI_Comm_accept/MPI_Comm_connect 

I followed the example code here: 
http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/node98.htm#Node98 
http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/node106.htm 
and was able to get them working if I use mpiexec / smpd. However, I want to start the executable without using mpiexec / smpd since we already have our own windows service application to deal with compute node resource allocation. 

I already read through a lot of documentations, but couldn't find my answers to some of the basic problems that I am trying to solve: 

1. Base on the MPI 2.2 document section 8.8 here: 
http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf 
It's not a requirement to use mpiexec / mpirun to start an MPI program. 
Is there any example available for starting the MPI program without using mpiexec / mpirun? 

2. I found an example that allows debugging without the use of a process manager by setting environment variable in section 9.8 of this doc: 
http://www.mcs.anl.gov/research/projects/mpich2/documentation/files/mpich2-1.3.2-windevguide.pdf 
It noted that this cannot support MPI_Comm_Spawn. Is that a technical limitation or there is a workaround? 

Thanks in advance for your help! 
Eric _______________________________________________ 
mpich-discuss mailing list 
mpich-discuss at mcs.anl.gov 
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss 