Please bear with me, it is a long query.<br><br>I don't think those instructions are particularly useful to me (see Rajeev's reply below). First of all, I cannot use 'srun' from command line, I can only use it as an option to mpirun when I am submitting the job through LSF. What I mean is, when I use srun from command line, this is what I get (the command is from the script mentioned at the bottom of the webpage you provided, Rajeev):<br>
<br>[So What?? ~]$ srun hostname -s | sort -u<br>srun: error: Unable to allocate resources: No partition specified or system default partition<br><br>But when I submit it through LSF, this is what I get:<br>[So What?? ~]$ bsub -n4 -o srun.%J.out mpirun -srun hostname -s | sort -u<br>
Job <14474> is submitted to default queue <normal>.<br><br><output><br>Your job looked like:<br><br>------------------------------------------------------------<br># LSBATCH: User input<br>mpirun -srun hostname -s<br>
------------------------------------------------------------<br><br>Successfully completed.<br><br>Resource usage summary:<br><br> CPU time : 0.14 sec.<br> Max Memory : 2 MB<br> Max Swap : 103 MB<br>
<br><br>The output (if any) follows:<br><br>n4<br>n4<br>n4<br>n4<br></output><br clear="all"><br>Now this is true when I am using HP-MPI. When I switch to MPICH1, the output is like this:<br>[So What?? ~]$ bsub -n15 -o srun.%J.out mpirun -srun -np 15 -machinefile mpd.hosts hostname<br>
Job <14479> is submitted to default queue <normal>.<br><br><output><br>Your job looked like:<br><br>------------------------------------------------------------<br># LSBATCH: User input<br>mpirun -srun -np 15 -machinefile mpd.hosts hostname<br>
------------------------------------------------------------<br><br>Exited with exit code 1.<br><br>Resource usage summary:<br><br> CPU time : 0.12 sec.<br> Max Memory : 2 MB<br> Max Swap : 103 MB<br>
<br><br>The output (if any) follows:<br><br>Warning: Command line arguments for program should be given<br>after the program name. Assuming that hostname is a<br>command line argument for the program.<br>Missing: program name<br>
Program -srun either does not exist, is not<br>executable, or is an erroneous argument to mpirun.<br></output><br><br>The SLURM version that we are using here is:<br>[So What?? ~]$ srun --version<br>slurm 1.0.15<br>
<br>That means, the patch that website mentions for SLURM and MPICH1 combo doesn't apply here as it is for SLURM version 1.2.11 of higher.<br><br>If I go to MPICH2 and use it through bsub, it obviously fails, probably because it wasn't configured with the options that Rajeev had suggested earlier.<br>
<br>The problem boils down to this:<br>1. The cluster is NOT configured for users to access each node individually, it's forbidden. I cannot launch my tasks (including starting mpd) on any node different from the head node.<br>
2. This is so done as to prevent users from ssh-ing to individual nodes and submitting jobs, thereby hogging resources. Users can only submit jobs to other nodes via LSF (i.e. when bsub [options] mpirun -srun ./executable is used).<br>
3. Obviously, since only HP-MPI impelemtation allows mpirun to take srun option while used with bsub, only in that implemetation, can I get my programs to run on multiple nodes.<br><br>So, it is not just MPICH+SLURM that I need, I also need help with MPICH+(LSF+SLURM).<br>
<br>Hail your patience.<br><br>Gauri.<br>---------<br><br>
Date: Wed, 25 Feb 2009 12:34:29 -0600<br>
From: "Rajeev Thakur" <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>><br>
Subject: Re: [mpich-discuss] <span class="il">HP</span>-<span class="il">XC</span> 3000 cluster issues<br>
To: <<a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a>><br>
Message-ID: <<a href="mailto:9273167066F94A0391E51E21D045C0FB@mcs.anl.gov">9273167066F94A0391E51E21D045C0FB@mcs.anl.gov</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Gauri,<br>
For MPICH-1, the instructions at the bottom of<br>
<a href="https://computing.llnl.gov/linux/slurm/quickstart.html" target="_blank">https://computing.llnl.gov/linux/slurm/quickstart.html</a> may be sufficient (I<br>
don't know).<br>
<br>
Rajeev<br>
<br>
<br>
<br>