No, I didn't know all this! But yes, it does make sense now. Thanks so much. Our cluster has slurm version 1.0.15 installed on it. From what I can gather from this page (<a href="https://computing.llnl.gov/linux/slurm/news.html">https://computing.llnl.gov/linux/slurm/news.html</a>), the mpiexec wrapper for SLURM is available for versions 1.2 onwards. I will try to install SLURM 1.2 and see if we can get it to work. Thanks again.<br>
<br>Gauri.<br>---------<br>
<br><br><div class="gmail_quote">On Thu, Apr 2, 2009 at 11:37 AM, Pavan Balaji <span dir="ltr"><<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Gauri,<br>
<br>
This might be a repetition of things you already know, but here's some brief description of the relationship between PMI and the process manager (each process manager has its own mpiexec).<br>
<br>
MPI applications talk to process managers to launch them as well as get information such as their rank, the size of the job, etc. The protocol they use to do this communication is called PMI. Unfortunately, everyone went out and implemented their own PMI library, e.g., (a) simple PMI (MPICH2's default PMI library), (b) smpd PMI (for linux/windows compatibility; will be deprecated soon) and (c) slurm PMI (implemented by the slurm guys).<br>
<br>
MPD, Gforker, Remshell, Hydra, OSC mpiexec, OSU mpirun and probably many other process managers use the simple PMI wire protocol. So, as long as the MPI application is linked with the simple PMI library, you can use any of these process managers interchangeably. Simple PMI library is what you are linked to by default when you build MPICH2 using the default options.<br>
<br>
srun uses slurm PMI. When you configure MPICH2 using --with-pmi=slurm, it should link with the slurm PMI library (I didn't go and check the code, but I think this is correct). Only srun is compatible with this slurm PMI library, so only that can be used. Note that the slurm folks came out with their own "mpiexec" executable, which essentially wraps around srun, so that uses the slurm PMI as well and should work. But I've never used it.<br>
<br>
So, in some sense, mpiexec or srun is just a user interface for you to talk in the appropriate PMI wire protocol. If you have a mismatch, the MPI process will not be able to detect their rank, the job size, etc., so all processes think they are rank 0 (this is what you are seeing).<br>
<br>
What is your exact requirement? If your requirement is to use the "mpiexec" interface, you can either use MPD (which will require MPICH2's default configuration), or try to download slurm's mpiexec (which will require MPICH2 to be configured using --with-slurm=<foo>).<br>
<br>
On the other hand, if your requirement is to use slurm, but don't care about the actual user interface, you can configure MPICH2 using --with-slurm=<foo> and just use srun.<br>
<br>
As a final note, we are working on defining PMI-2, which will hopefully unify all this into one single library and these issues will be gone.<br>
<br>
Hope that helps.<br>
<br>
-- Pavan<br>
<br>
Gauri Kulkarni wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im">
I cannot. Because I get the SAME errors. Following is the LSF script I used to launch the job.<br>
<br>
#!/bin/bash<br>
#BSUB -L /bin/bash<br>
#BSUB -n 8<br>
#BSUB -N<br>
#BSUB -o /data1/visitor/cgaurik/testmpi/helloworld.mympi.mpiexec.%J.out<br>
<br>
cd /data1/visitor/cgaurik/testmpi<br>
/data1/visitor/cgaurik/mympi/bin/mpiexec -np 8 ./helloworld.mympi<br>
<br>
The job is NOT parallelized i.e. every process is rank 0. And errors are same. Of course, if I change (as, I think, Dave pointed out) to srun ./helloworld.mympi in the last line of the script, everything is all rosy. My question is, (maybe it's obvious...) that if my mpich2 is configured with the options "--with-pmi=slurm --with-pm=no --with-slurm=/path/to/slurm/lib", can I still use mpiexec?<br>
<br>
Gauri.<br>
---------<br>
<br>
<br></div><div class="im">
On Wed, Apr 1, 2009 at 11:42 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov" target="_blank">thakur@mcs.anl.gov</a> <mailto:<a href="mailto:thakur@mcs.anl.gov" target="_blank">thakur@mcs.anl.gov</a>>> wrote:<br>
<br>
You need to use the mpicc and mpiexec from the MPICH2 installation<br>
that was built to use MPD.<br>
Rajeev<br>
<br>
------------------------------------------------------------------------<br>
*From:* <a href="mailto:mpich-discuss-bounces@mcs.anl.gov" target="_blank">mpich-discuss-bounces@mcs.anl.gov</a><br>
<mailto:<a href="mailto:mpich-discuss-bounces@mcs.anl.gov" target="_blank">mpich-discuss-bounces@mcs.anl.gov</a>><br>
[mailto:<a href="mailto:mpich-discuss-bounces@mcs.anl.gov" target="_blank">mpich-discuss-bounces@mcs.anl.gov</a><br>
<mailto:<a href="mailto:mpich-discuss-bounces@mcs.anl.gov" target="_blank">mpich-discuss-bounces@mcs.anl.gov</a>>] *On Behalf Of *Gauri<br>
Kulkarni<br>
*Sent:* Wednesday, April 01, 2009 8:56 AM<br></div>
*To:* <a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a> <mailto:<a href="mailto:mpich-discuss@mcs.anl.gov" target="_blank">mpich-discuss@mcs.anl.gov</a>><div><div></div>
<div class="h5"><br>
*Subject:* [mpich-discuss] What do these errors mean?<br>
<br>
Hi,<br>
<br>
I am using MPICH2-1.0.7 (I cannot go to 1.0.8 right now) which<br>
is configured to be used with SLURM. That is, the process<br>
manager is SLURM and NOT mpd. When I submit my job through bsub<br>
(bsub [options] srun ./helloworld.mympi), it works perfectly. I<br>
cannot use mpiexec as it is not the one spawning jobs, I must<br>
use srun. My question is, can I still use mpiexec from<br>
command-line? Well.. I tried. Here is the output:<br>
<br>
mpiexec -n 2 ./helloworld.mympi<br>
mpiexec_n53: cannot connect to local mpd<br>
(/tmp/mpd2.console_cgaurik); possible causes:<br>
1. no mpd is running on this host<br>
2. an mpd is running but was started without a "console" (-n<br>
option)<br>
In case 1, you can start an mpd on this host with:<br>
mpd &<br>
and you will be able to run jobs just on this host.<br>
For more details on starting mpds on a set of hosts, see<br>
the MPICH2 Installation Guide.<br>
<br>
Then:<br>
<br>
mpd &<br>
mpiexec -n 2 ./helloworld.mympi<br>
<br>
*Hello world! I'm 0 of 2 on n53*<br>
Fatal error in MPI_Finalize: Other MPI error, error stack:<br>
MPI_Finalize(255)...................: MPI_Finalize failed<br>
MPI_Finalize(154)...................:<br>
MPID_Finalize(94)...................:<br>
MPI_Barrier(406)....................:<br>
MPI_Barrier(comm=0x44000002) failed<br>
MPIR_Barrier(77)....................:<br>
MPIC_Sendrecv(120)..................:<br>
MPID_Isend(103).....................: failure occurred while<br>
attempting to send an eager message<br>
MPIDI_CH3_iSend(172)................:<br>
MPIDI_CH3I_VC_post_sockconnect(1090):<br>
MPIDI_PG_SetConnInfo(615)...........: PMI_KVS_Get failedFatal<br>
error in MPI_Finalize: Other MPI error, error stack:<br>
MPI_Finalize(255)...................: MPI_Finalize failed<br>
MPI_Finalize(154)...................:<br>
MPID_Finalize(94)...................:<br>
MPI_Barrier(406)....................:<br>
MPI_Barrier(comm=0x44000002) failed<br>
MPIR_Barrier(77)....................:<br>
MPIC_Sendrecv(120)..................:<br>
MPID_Isend(103).....................: failure occurred while<br>
attempting to send an eager message<br></div></div>
MP*Hello world! I'm 1 of 2 on n53*<div class="im"><br>
IDI_CH3_iSend(172)................:<br>
MPIDI_CH3I_VC_post_sockconnect(1090):<br>
MPIDI_PG_SetConnInfo(615)...........: PMI_KVS_Get failed<br>
<br>
The bold text shows that the job gets executed but there is a<br>
lot of other garbage. It seems to me that I can either configure<br>
MPICH2 to be used with cluster job scheduler or to be used from<br>
command line. I cannot have both.<br>
<br>
Am I right?<br>
<br>
-Gauri.<br>
----------<br>
<br>
<br>
</div></blockquote><font color="#888888">
<br>
-- <br>
Pavan Balaji<br>
<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
</font></blockquote></div><br>