[mpich-discuss] Hydra unable to execute jobs that use more than one node(host) under PBS RMK
Pavan Balaji
balaji at mcs.anl.gov
Thu Jan 14 04:52:45 CST 2010
Can you try the following two options:
1. Allocate a 2-node partition in interactive mode. Suppose the nodes
allocated to you (cat $PBS_NODEFILE) are n0 and n1 run the following
program:
% mpiexec.hydra -bootstrap fork -n 1 ssh -vvv n0 hostname : -n 1 ssh n1
/bin/true
... and send me the output for this (assuming that it shows the error
you reported)? Note that the above command does not have -rmk pbs in it.
2. Apply the attached patch to mpich2, and recompile it. Then try
running your application as:
% mpiexec.hydra -rmk pbs -verbose hostname
-- Pavan
On 01/14/2010 04:26 AM, Mário Costa wrote:
> Thanks for your reply!
>
> On Thu, Jan 14, 2010 at 1:37 AM, Pavan Balaji <balaji at mcs.anl.gov> wrote:
>> The error itself seems to be thrown by ssh, not Hydra. Based on some
>> googling, this seems to be a common problem with host-based
>> authentication in ssh. For example, see
>> https://www.cs.uwaterloo.ca/twiki/view/CF/SSHHostBasedAuthentication
>
> You are right, host-based authentication is not setup only publickey.
>
>> Can someone check this on your system (my guess is that something is
>> wrong with nodes 125 and 126, if it helps)? Alternatively, can you setup
>> a key based ssh (either by using a passwordless key, or an ssh agent) to
>> work around this?
>
> No problem with ssh to those nodes, if I do it by hand (ssh using
> publickey authentication, I forgot to mention in the test) it works
> properly.
>
> Could it be that hydra is forcing somehow host-based authentication?
>
>> Note that though both hydra and mpd use ssh, they use different models,
>> so which node ssh's to which other node will be different with both the
>> process managers.
>
> I wrote some wrapping scripts to create dynamic mpd rings spanning
> only PBS assigned nodes to the job, in this sense I think it would be
> similar!?
> All nodes can ssh to each other properly using publickey.
>
> If you need some additional info let me know.
>
> Regards,
> Mário
>> -- Pavan
>>
>> On 01/13/2010 05:48 PM, Mário Costa wrote:
>>> Hello,
>>>
>>> I'm currently testing mpiexec.hydra under PBS(Torque) in Enterprise
>>> Suse 10, using the default bootstrap server (ssh).
>>>
>>> With MPD, I've managed to execute successfully jobs using any number
>>> of nodes(hosts)/processors.
>>> I've setup ssh keys, known_hosts, ... i've been using a wrapper script
>>> to manage mpi rings complying to the PBS provided nodes/resources, to
>>> execute under PBS...
>>>
>>> With Hydra, I successfully managed to execute jobs that span over one
>>> node only, tested it with four processors and less.
>>>
>>> My test, a shell script:
>>>
>>> #!/bin/bash
>>> env | grep PMI
>>>
>>> When I submit a job that spans over more than one node I get the
>>> following errors.
>>>
>>> 1. Job hangs till its killed by PBS due to exceeded time limit, used 3
>>> nodes, 8 procs.
>>>
>>> stderr:
>>>
>>> bad fd
>>> ssh_keysign: no reply
>>> key_sign failed
>>> bad fd
>>> ssh_keysign: no reply
>>> key_sign failed
>>> =>> PBS: job killed: walltime 917 exceeded limit 900
>>> Killed by signal 15.
>>> Killed by signal 15.
>>>
>>> stdout:
>>>
>>> PMI_PORT=gorgon127:35454
>>> PMI_ID=0
>>> PMI_PORT=gorgon127:35454
>>> PMI_ID=1
>>> PMI_PORT=gorgon127:35454
>>> PMI_ID=2
>>>
>>> ps at the node where PBS executed the script that invoked hydra:
>>>
>>> userX 10187 0.0 0.0 3928 460 ? S 23:03 0:00 pbs_demux
>>> userX 10205 0.0 0.0 9360 1548 ? S 23:03 0:00
>>> /bin/bash /var/spool/torque/mom_priv/jobs/9950.clulne.SC
>>> userX 10208 0.0 0.0 6072 764 ? S 23:03 0:00
>>> mpiexec.hydra -rmk pbs ./test.sh
>>> userX 10209 0.0 0.0 0 0 ? Z 23:03 0:00 [ssh] <defunct>
>>> userX 10210 0.0 0.0 24084 2508 ? S 23:03 0:00
>>> /usr/bin/ssh -x node126 /usr/bin/pmi_proxy --launch-mode 1
>>> --proxy-port node127:36669 --bootstrap ssh --partition-id 1
>>> userX 10211 0.0 0.0 24088 2508 ? S 23:03 0:00
>>> /usr/bin/ssh -x node125 /usr/bin/pmi_proxy --launch-mode 1
>>> --proxy-port node127:36669 --bootstrap ssh --partition-id 2
>>> userX 10215 0.0 0.0 0 0 ? Z 23:03 0:00
>>> [ssh-keysign] <defunct>
>>> userX 10255 0.0 0.0 0 0 ? Z 23:03 0:00
>>> [ssh-keysign] <defunct>
>>> root 10256 0.1 0.0 43580 3520 ? Ss 23:04 0:00 sshd:
>>> userX [priv]
>>> userX 10258 0.0 0.0 43580 1968 ? S 23:04 0:00 sshd:
>>> userX at pts/0
>>>
>>> 2. Job terminated immediately, used 2 nodes, 8 procs (but other test
>>> with 2 nodes had the same result as above).
>>>
>>> stderr:
>>>
>>> bad fd
>>> ssh_keysign: no reply
>>> key_sign failed
>>> Disconnecting: Bad packet length 232220199.
>>>
>>> stdout:
>>>
>>> PMI_PORT=gorgon116:52217
>>> PMI_ID=1
>>> PMI_PORT=gorgon116:52217
>>> PMI_ID=3
>>> PMI_PORT=gorgon116:52217
>>> PMI_ID=2
>>> PMI_PORT=gorgon116:52217
>>> PMI_ID=0
>>>
>>> Any idea of what might be wrong ?
>>>
>>> There is something wrong with ssh, in test 1, I've ssh to the node and
>>> executed the command showed in the ps, and it executed properly with
>>> the respective partion/PMI_IDs being dysplayed.
>>>
>>> Since I've managed to use MPD with any kind of problems I would
>>> presume my ssh is working properly.
>>>
>>> Could it be that the is something wrong with hydra ?
>>>
>>> Thanks, regards,
>>>
>>> Mário
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> --
>> Pavan Balaji
>> http://www.mcs.anl.gov/~balaji
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hydra-verbose.patch
Type: text/x-patch
Size: 566 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20100114/672de955/attachment-0001.bin>
More information about the mpich-discuss
mailing list