[mpich-discuss] mpdboot failure

Tue Nov 18 16:09:48 CST 2008

As a follow up, here is an example of the error message
in /var/log/messages:

Nov 18 14:08:53 bduc-i32-6 mpd: mpd starting; no mpdid yet
Nov 18 14:08:53 bduc-i32-6 mpd: mpd has mpdid=bduc-i32-6_37197
(port=37197)
Nov 18 14:08:53 bduc-i32-6 mpd: bduc-i32-6_37197 (connect_lhs 900):
failed to connect to lhs at bduc-i32-4 41311
Nov 18 14:08:53 bduc-i32-6 mpd: bduc-i32-6_37197 (enter_ring 855): lhs
connect failed
Nov 18 14:08:53 bduc-i32-6 mpd: bduc-i32-6_37197 (run 252): failed to
enter ring
Nov 18 14:08:53 bduc-i32-6 mpd: mpd ending mpdid=bduc-i32-6_37197
(inside cleanup)

Very Respectfully,
Cesar Covarrubias

On Tue, 2008-11-18 at 11:14 -0800, Cesar Covarrubias wrote:
> Hello,
> 
> We are still having trouble getting mpd running on a large number of our
> nodes. The error we get is mpdboot_bduc-sched.nacs.uci.edu
> (handle_mpd_output 392): failed to handshake with mpd on bduc-i32-6;
> recvd output={}. The problem is that it the node it chooses to fail on
> is random. It is not always the same node. All nodes have
> the /etc/mpd.conf file chmod'ed to 600. mpd is not running on any of the
> nodes. The mpdcheck returns no errors (as you can see by the output
> below). All the nodes are on a private subnet.
> 
> Ideas as to why this is failing?
> 
> Very Respectfully,
> Cesar Covarrubias
> 
> -bash-3.2# mpdcheck -f mpd.hosts -ssh -v
> obtaining hostname via gethostname and getfqdn
> gethostname gives  bduc-sched.nacs.uci.edu
> getfqdn gives  bduc-sched.nacs.uci.edu
> checking out unqualified hostname; make sure is not "localhost", etc.
> checking out qualified hostname; make sure is not "localhost", etc.
> obtain IP addrs via qualified and unqualified hostnames;  make sure
> other than 127.0.0.1
> gethostbyname_ex:  ('bduc-sched.nacs.uci.edu', ['bduc-sched'],
> ['128.200.15.19'])
> gethostbyname_ex:  ('bduc-sched.nacs.uci.edu', ['bduc-sched'],
> ['128.200.15.19'])
> checking that IP addrs resolve to same host
> now do some gethostbyaddr and gethostbyname_ex for machines in hosts
> filechecking gethostbyXXX for unqualified bduc-i32-1
> gethostbyname_ex:  ('bduc-i32-1', [], ['192.168.0.10'])
> checking gethostbyXXX for qualified bduc-i32-1
> gethostbyname_ex:  ('bduc-i32-1', [], ['192.168.0.10'])
> checking gethostbyXXX for unqualified bduc-i32-3
> gethostbyname_ex:  ('bduc-i32-3', [], ['192.168.0.12'])
> checking gethostbyXXX for qualified bduc-i32-3
> gethostbyname_ex:  ('bduc-i32-3', [], ['192.168.0.12'])
> checking gethostbyXXX for unqualified bduc-i32-4
> gethostbyname_ex:  ('bduc-i32-4', [], ['192.168.0.13'])
> checking gethostbyXXX for qualified bduc-i32-4
> gethostbyname_ex:  ('bduc-i32-4', [], ['192.168.0.13'])
> checking gethostbyXXX for unqualified bduc-i32-5
> gethostbyname_ex:  ('bduc-i32-5', [], ['192.168.0.14'])
> checking gethostbyXXX for qualified bduc-i32-5
> gethostbyname_ex:  ('bduc-i32-5', [], ['192.168.0.14'])
> checking gethostbyXXX for unqualified bduc-i32-6
> gethostbyname_ex:  ('bduc-i32-6', [], ['192.168.0.15'])
> checking gethostbyXXX for qualified bduc-i32-6
> gethostbyname_ex:  ('bduc-i32-6', [], ['192.168.0.15'])
> checking gethostbyXXX for unqualified bduc-i32-7
> gethostbyname_ex:  ('bduc-i32-7', [], ['192.168.0.16'])
> checking gethostbyXXX for qualified bduc-i32-7
> gethostbyname_ex:  ('bduc-i32-7', [], ['192.168.0.16'])
> checking gethostbyXXX for unqualified bduc-i32-8
> gethostbyname_ex:  ('bduc-i32-8', [], ['192.168.0.17'])
> checking gethostbyXXX for qualified bduc-i32-8
> gethostbyname_ex:  ('bduc-i32-8', [], ['192.168.0.17'])
> checking gethostbyXXX for unqualified bduc-i32-9
> gethostbyname_ex:  ('bduc-i32-9', [], ['192.168.0.18'])
> checking gethostbyXXX for qualified bduc-i32-9
> gethostbyname_ex:  ('bduc-i32-9', [], ['192.168.0.18'])
> checking gethostbyXXX for unqualified bduc-i32-10
> gethostbyname_ex:  ('bduc-i32-10', [], ['192.168.0.19'])
> checking gethostbyXXX for qualified bduc-i32-10
> gethostbyname_ex:  ('bduc-i32-10', [], ['192.168.0.19'])
> checking gethostbyXXX for unqualified bduc-i32-11
> gethostbyname_ex:  ('bduc-i32-11', [], ['192.168.0.20'])
> checking gethostbyXXX for qualified bduc-i32-11
> gethostbyname_ex:  ('bduc-i32-11', [], ['192.168.0.20'])
> checking gethostbyXXX for unqualified bduc-i32-12
> gethostbyname_ex:  ('bduc-i32-12', [], ['192.168.0.21'])
> checking gethostbyXXX for qualified bduc-i32-12
> gethostbyname_ex:  ('bduc-i32-12', [], ['192.168.0.21'])
> checking gethostbyXXX for unqualified bduc-i32-13
> gethostbyname_ex:  ('bduc-i32-13', [], ['192.168.0.22'])
> checking gethostbyXXX for qualified bduc-i32-13
> gethostbyname_ex:  ('bduc-i32-13', [], ['192.168.0.22'])
> checking gethostbyXXX for unqualified bduc-i32-14
> gethostbyname_ex:  ('bduc-i32-14', [], ['192.168.0.23'])
> checking gethostbyXXX for qualified bduc-i32-14
> gethostbyname_ex:  ('bduc-i32-14', [], ['192.168.0.23'])
> checking gethostbyXXX for unqualified bduc-i32-15
> gethostbyname_ex:  ('bduc-i32-15', [], ['192.168.0.24'])
> checking gethostbyXXX for qualified bduc-i32-15
> gethostbyname_ex:  ('bduc-i32-15', [], ['192.168.0.24'])
> checking gethostbyXXX for unqualified bduc-i32-16
> gethostbyname_ex:  ('bduc-i32-16', [], ['192.168.0.25'])
> checking gethostbyXXX for qualified bduc-i32-16
> gethostbyname_ex:  ('bduc-i32-16', [], ['192.168.0.25'])
> checking gethostbyXXX for unqualified bduc-i32-17
> gethostbyname_ex:  ('bduc-i32-17', [], ['192.168.0.26'])
> checking gethostbyXXX for qualified bduc-i32-17
> gethostbyname_ex:  ('bduc-i32-17', [], ['192.168.0.26'])
> checking gethostbyXXX for unqualified bduc-i32-18
> gethostbyname_ex:  ('bduc-i32-18', [], ['192.168.0.27'])
> checking gethostbyXXX for qualified bduc-i32-18
> gethostbyname_ex:  ('bduc-i32-18', [], ['192.168.0.27'])
> checking gethostbyXXX for unqualified bduc-i32-19
> gethostbyname_ex:  ('bduc-i32-19', [], ['192.168.0.28'])
> checking gethostbyXXX for qualified bduc-i32-19
> gethostbyname_ex:  ('bduc-i32-19', [], ['192.168.0.28'])
> checking gethostbyXXX for unqualified bduc-i32-20
> gethostbyname_ex:  ('bduc-i32-20', [], ['192.168.0.29'])
> checking gethostbyXXX for qualified bduc-i32-20
> gethostbyname_ex:  ('bduc-i32-20', [], ['192.168.0.29'])
> checking gethostbyXXX for unqualified bduc-i32-21
> gethostbyname_ex:  ('bduc-i32-21', [], ['192.168.0.30'])
> checking gethostbyXXX for qualified bduc-i32-21
> gethostbyname_ex:  ('bduc-i32-21', [], ['192.168.0.30'])
> checking gethostbyXXX for unqualified bduc-i32-22
> gethostbyname_ex:  ('bduc-i32-22', [], ['192.168.0.31'])
> checking gethostbyXXX for qualified bduc-i32-22
> gethostbyname_ex:  ('bduc-i32-22', [], ['192.168.0.31'])
> checking gethostbyXXX for unqualified bduc-i32-24
> gethostbyname_ex:  ('bduc-i32-24', [], ['192.168.0.33'])
> checking gethostbyXXX for qualified bduc-i32-24
> gethostbyname_ex:  ('bduc-i32-24', [], ['192.168.0.33'])
> checking gethostbyXXX for unqualified bduc-i32-25
> gethostbyname_ex:  ('bduc-i32-25', [], ['192.168.0.34'])
> checking gethostbyXXX for qualified bduc-i32-25
> gethostbyname_ex:  ('bduc-i32-25', [], ['192.168.0.34'])
> checking gethostbyXXX for unqualified bduc-i32-27
> gethostbyname_ex:  ('bduc-i32-27', [], ['192.168.0.36'])
> checking gethostbyXXX for qualified bduc-i32-27
> gethostbyname_ex:  ('bduc-i32-27', [], ['192.168.0.36'])
> checking gethostbyXXX for unqualified bduc-i32-28
> gethostbyname_ex:  ('bduc-i32-28', [], ['192.168.0.37'])
> checking gethostbyXXX for qualified bduc-i32-28
> gethostbyname_ex:  ('bduc-i32-28', [], ['192.168.0.37'])
> checking gethostbyXXX for unqualified bduc-i32-29
> gethostbyname_ex:  ('bduc-i32-29', [], ['192.168.0.38'])
> checking gethostbyXXX for qualified bduc-i32-29
> gethostbyname_ex:  ('bduc-i32-29', [], ['192.168.0.38'])
> checking gethostbyXXX for unqualified bduc-i32-30
> gethostbyname_ex:  ('bduc-i32-30', [], ['192.168.0.39'])
> checking gethostbyXXX for qualified bduc-i32-30
> gethostbyname_ex:  ('bduc-i32-30', [], ['192.168.0.39'])
> checking gethostbyXXX for unqualified bduc-i32-31
> gethostbyname_ex:  ('bduc-i32-31', [], ['192.168.0.40'])
> checking gethostbyXXX for qualified bduc-i32-31
> gethostbyname_ex:  ('bduc-i32-31', [], ['192.168.0.40'])
> checking gethostbyXXX for unqualified bduc-i32-32
> gethostbyname_ex:  ('bduc-i32-32', [], ['192.168.0.41'])
> checking gethostbyXXX for qualified bduc-i32-32
> gethostbyname_ex:  ('bduc-i32-32', [], ['192.168.0.41'])
> checking gethostbyXXX for unqualified bduc-i32-33
> gethostbyname_ex:  ('bduc-i32-33', [], ['192.168.0.42'])
> checking gethostbyXXX for qualified bduc-i32-33
> gethostbyname_ex:  ('bduc-i32-33', [], ['192.168.0.42'])
> checking gethostbyXXX for unqualified bduc-i32-34
> gethostbyname_ex:  ('bduc-i32-34', [], ['192.168.0.43'])
> checking gethostbyXXX for qualified bduc-i32-34
> gethostbyname_ex:  ('bduc-i32-34', [], ['192.168.0.43'])
> checking gethostbyXXX for unqualified bduc-i32-35
> gethostbyname_ex:  ('bduc-i32-35', [], ['192.168.0.44'])
> checking gethostbyXXX for qualified bduc-i32-35
> gethostbyname_ex:  ('bduc-i32-35', [], ['192.168.0.44'])
> checking gethostbyXXX for unqualified bduc-i32-36
> gethostbyname_ex:  ('bduc-i32-36', [], ['192.168.0.45'])
> checking gethostbyXXX for qualified bduc-i32-36
> gethostbyname_ex:  ('bduc-i32-36', [], ['192.168.0.45'])
> checking gethostbyXXX for unqualified bduc-i32-37
> gethostbyname_ex:  ('bduc-i32-37', [], ['192.168.0.46'])
> checking gethostbyXXX for qualified bduc-i32-37
> gethostbyname_ex:  ('bduc-i32-37', [], ['192.168.0.46'])
> checking gethostbyXXX for unqualified bduc-i32-38
> gethostbyname_ex:  ('bduc-i32-38', [], ['192.168.0.47'])
> checking gethostbyXXX for qualified bduc-i32-38
> gethostbyname_ex:  ('bduc-i32-38', [], ['192.168.0.47'])
> checking gethostbyXXX for unqualified bduc-i32-39
> gethostbyname_ex:  ('bduc-i32-39', [], ['192.168.0.48'])
> checking gethostbyXXX for qualified bduc-i32-39
> gethostbyname_ex:  ('bduc-i32-39', [], ['192.168.0.48'])
> checking gethostbyXXX for unqualified bduc-i32-40
> gethostbyname_ex:  ('bduc-i32-40', [], ['192.168.0.49'])
> checking gethostbyXXX for qualified bduc-i32-40
> gethostbyname_ex:  ('bduc-i32-40', [], ['192.168.0.49'])
> trying: ssh bduc-i32-1 -x -n /bin/echo hello
> trying: ssh bduc-i32-3 -x -n /bin/echo hello
> trying: ssh bduc-i32-4 -x -n /bin/echo hello
> trying: ssh bduc-i32-5 -x -n /bin/echo hello
> trying: ssh bduc-i32-6 -x -n /bin/echo hello
> trying: ssh bduc-i32-7 -x -n /bin/echo hello
> trying: ssh bduc-i32-8 -x -n /bin/echo hello
> trying: ssh bduc-i32-9 -x -n /bin/echo hello
> trying: ssh bduc-i32-10 -x -n /bin/echo hello
> trying: ssh bduc-i32-11 -x -n /bin/echo hello
> trying: ssh bduc-i32-12 -x -n /bin/echo hello
> trying: ssh bduc-i32-13 -x -n /bin/echo hello
> trying: ssh bduc-i32-14 -x -n /bin/echo hello
> trying: ssh bduc-i32-15 -x -n /bin/echo hello
> trying: ssh bduc-i32-16 -x -n /bin/echo hello
> trying: ssh bduc-i32-17 -x -n /bin/echo hello
> trying: ssh bduc-i32-18 -x -n /bin/echo hello
> trying: ssh bduc-i32-19 -x -n /bin/echo hello
> trying: ssh bduc-i32-20 -x -n /bin/echo hello
> trying: ssh bduc-i32-21 -x -n /bin/echo hello
> trying: ssh bduc-i32-22 -x -n /bin/echo hello
> trying: ssh bduc-i32-24 -x -n /bin/echo hello
> trying: ssh bduc-i32-25 -x -n /bin/echo hello
> trying: ssh bduc-i32-27 -x -n /bin/echo hello
> trying: ssh bduc-i32-28 -x -n /bin/echo hello
> trying: ssh bduc-i32-29 -x -n /bin/echo hello
> trying: ssh bduc-i32-30 -x -n /bin/echo hello
> trying: ssh bduc-i32-31 -x -n /bin/echo hello
> trying: ssh bduc-i32-32 -x -n /bin/echo hello
> trying: ssh bduc-i32-33 -x -n /bin/echo hello
> trying: ssh bduc-i32-34 -x -n /bin/echo hello
> trying: ssh bduc-i32-35 -x -n /bin/echo hello
> trying: ssh bduc-i32-36 -x -n /bin/echo hello
> trying: ssh bduc-i32-37 -x -n /bin/echo hello
> trying: ssh bduc-i32-38 -x -n /bin/echo hello
> trying: ssh bduc-i32-39 -x -n /bin/echo hello
> trying: ssh bduc-i32-40 -x -n /bin/echo hello
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-1 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 43076
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-3 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 38649
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-4 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 40831
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-5 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 57153
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-6 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 60407
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-7 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 52646
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-8 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 47523
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-9 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 50172
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-10 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 51821
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-11 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 36082
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-12 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 57250
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-13 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 47474
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-14 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 51971
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-15 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 54481
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-16 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 49851
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-17 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 33898
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-18 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 56621
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-19 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 37119
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-20 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 38601
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-21 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 43570
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-22 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 42130
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-24 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 41582
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-25 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 50515
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-27 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 48864
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-28 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 47361
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-29 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 44604
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-30 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 41417
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-31 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 48330
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-32 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 47243
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-33 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 35428
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-34 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 59147
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-35 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 54136
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-36 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 56116
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-37 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 59475
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-38 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 52834
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-39 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 58080
> starting server: /sge62/mpich2/bin/mpdcheck.py -s
> starting client: ssh bduc-i32-40 -x -n /sge62/mpich2/bin/mpdcheck.py -c
> bduc-sched.nacs.uci.edu 57115
> 
> 