================================================================================================== mpiexec options: ---------------- Base path: /home/kkharba/rMPI-workspace/hydra-install/bin/ Bootstrap server: (null) Debug level: 1 Enable X: -1 Global environment: ------------------- HOSTNAME=opt09 TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=192.168.0.100 37164 22 CVSROOT=kkharba@optout.csc.ncsu.edu:/home/cvs/cvs QTDIR=/usr/lib64/qt-3.3 QTINC=/usr/lib64/qt-3.3/include SSH_TTY=/dev/pts/6 USER=kkharba LD_LIBRARY_PATH=/home/kkharba/rMPI-workspace/hydra-install/lib:/home/kkharba/blcr_intro/blcr-install-0.8.3/lib:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/lib:/home/kkharba/blcr_intro/mrmpi-install/lib: LS_COLORS=rs RSHCOMMAND=ssh -q MAIL=/var/spool/mail/kkharba PATH=/home/kkharba/rMPI-workspace/hydra-install/bin:/home/kkharba/blcr_intro/blcr-install-0.8.3/bin:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/kkharba/bin C_INCLUDE_PATH=/home/kkharba/rMPI-workspace/hydra-install/include:/home/kkharba/blcr_intro/blcr-install-0.8.3/include:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/include: PWD=/home/kkharba/rMPI-workspace/mpich2-1.3rc2 LANG=en_US.UTF-8 MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles LOADEDMODULES=(null) SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass HISTCONTROL=ignoreboth SHLVL=1 HOME=/home/kkharba LOGNAME=kkharba QTLIB=/usr/lib64/qt-3.3/lib CVS_RSH=ssh SSH_CONNECTION=192.168.0.100 37164 192.168.0.109 22 MODULESHOME=/usr/share/Modules LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 module=() { eval `/usr/bin/modulecmd bash $*` } _=/home/kkharba/rMPI-workspace/hydra-install/bin/mpiexec OLDPWD=/home/kkharba Proxy information: ********************* Proxy ID: 1 ----------------- Proxy name: optmpi09 Process count: 1 Start PID: 0 Proxy exec list: .................... Exec: ./examples/cpi; Process count: 1 Proxy ID: 2 ----------------- Proxy name: optmpi10 Process count: 1 Start PID: 1 Proxy exec list: .................... Exec: ./examples/cpi; Process count: 1 ================================================================================================== [mpiexec@opt09] Timeout set to -1 (-1 means infinite) [mpiexec@opt09] Got a control port string of opt09:41229 Proxy launch args: /home/kkharba/rMPI-workspace/hydra-install/bin/hydra_pmi_proxy --control-port opt09:41229 --debug --demux poll --pgid 0 --enable-stdin 1 --proxy-id [mpiexec@opt09] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1 Arguments being passed to proxy 0: --version 1.3rc2 --interface-env-name MPICH_INTERFACE_NAME --hostname optmpi09 --global-core-count 2 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_5945_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 34 'HOSTNAME=opt09' 'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'SSH_CLIENT=192.168.0.100 37164 22' 'CVSROOT=kkharba@optout.csc.ncsu.edu:/home/cvs/cvs' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/6' 'USER=kkharba' 'LD_LIBRARY_PATH=/home/kkharba/rMPI-workspace/hydra-install/lib:/home/kkharba/blcr_intro/blcr-install-0.8.3/lib:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/lib:/home/kkharba/blcr_intro/mrmpi-install/lib:' 'LS_COLORS=rs' 'RSHCOMMAND=ssh -q' 'MAIL=/var/spool/mail/kkharba' 'PATH=/home/kkharba/rMPI-workspace/hydra-install/bin:/home/kkharba/blcr_intro/blcr-install-0.8.3/bin:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/kkharba/bin' 'C_INCLUDE_PATH=/home/kkharba/rMPI-workspace/hydra-install/include:/home/kkharba/blcr_intro/blcr-install-0.8.3/include:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/include:' 'PWD=/home/kkharba/rMPI-workspace/mpich2-1.3rc2' 'LANG=en_US.UTF-8' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'HISTCONTROL=ignoreboth' 'SHLVL=1' 'HOME=/home/kkharba' 'LOGNAME=kkharba' 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh' 'SSH_CONNECTION=192.168.0.100 37164 192.168.0.109 22' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1' 'module=() { eval `/usr/bin/modulecmd bash $*` }' '_=/home/kkharba/rMPI-workspace/hydra-install/bin/mpiexec' 'OLDPWD=/home/kkharba' --global-user-env 0 --global-system-env 0 --start-pid 0 --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/kkharba/rMPI-workspace/mpich2-1.3rc2 --exec-args 1 ./examples/cpi [mpiexec@opt09] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1 Arguments being passed to proxy 1: --version 1.3rc2 --interface-env-name MPICH_INTERFACE_NAME --hostname optmpi10 --global-core-count 2 --global-process-count 2 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_5945_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 34 'HOSTNAME=opt09' 'TERM=xterm' 'SHELL=/bin/bash' 'HISTSIZE=1000' 'SSH_CLIENT=192.168.0.100 37164 22' 'CVSROOT=kkharba@optout.csc.ncsu.edu:/home/cvs/cvs' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'SSH_TTY=/dev/pts/6' 'USER=kkharba' 'LD_LIBRARY_PATH=/home/kkharba/rMPI-workspace/hydra-install/lib:/home/kkharba/blcr_intro/blcr-install-0.8.3/lib:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/lib:/home/kkharba/blcr_intro/mrmpi-install/lib:' 'LS_COLORS=rs' 'RSHCOMMAND=ssh -q' 'MAIL=/var/spool/mail/kkharba' 'PATH=/home/kkharba/rMPI-workspace/hydra-install/bin:/home/kkharba/blcr_intro/blcr-install-0.8.3/bin:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/kkharba/bin' 'C_INCLUDE_PATH=/home/kkharba/rMPI-workspace/hydra-install/include:/home/kkharba/blcr_intro/blcr-install-0.8.3/include:/home/kkharba/rMPI-workspace/mpich2-1.3rc2-install/include:' 'PWD=/home/kkharba/rMPI-workspace/mpich2-1.3rc2' 'LANG=en_US.UTF-8' 'MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles' 'LOADEDMODULES=' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'HISTCONTROL=ignoreboth' 'SHLVL=1' 'HOME=/home/kkharba' 'LOGNAME=kkharba' 'QTLIB=/usr/lib64/qt-3.3/lib' 'CVS_RSH=ssh' 'SSH_CONNECTION=192.168.0.100 37164 192.168.0.109 22' 'MODULESHOME=/usr/share/Modules' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1' 'module=() { eval `/usr/bin/modulecmd bash $*` }' '_=/home/kkharba/rMPI-workspace/hydra-install/bin/mpiexec' 'OLDPWD=/home/kkharba' --global-user-env 0 --global-system-env 0 --start-pid 1 --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/kkharba/rMPI-workspace/mpich2-1.3rc2 --exec-args 1 ./examples/cpi [mpiexec@opt09] Launch arguments: /usr/bin/ssh -x optmpi09 /home/kkharba/rMPI-workspace/hydra-install/bin/hydra_pmi_proxy --control-port opt09:41229 --debug --demux poll --pgid 0 --enable-stdin 1 --proxy-id 0 [mpiexec@opt09] Launch arguments: /usr/bin/ssh -x optmpi10 /home/kkharba/rMPI-workspace/hydra-install/bin/hydra_pmi_proxy --control-port opt09:41229 --debug --demux poll --pgid 0 --enable-stdin 1 --proxy-id 1 [proxy:0:1@opt10] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:1@opt10] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:1@opt10] got pmi command (from 4): get_maxes [proxy:0:1@opt10] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:1@opt10] got pmi command (from 4): get_appnum [proxy:0:1@opt10] PMI response: cmd=appnum appnum=0 [proxy:0:1@opt10] got pmi command (from 4): get_my_kvsname [proxy:0:1@opt10] PMI response: cmd=my_kvsname kvsname=kvs_5945_0 [proxy:0:1@opt10] got pmi command (from 4): get_my_kvsname [proxy:0:1@opt10] PMI response: cmd=my_kvsname kvsname=kvs_5945_0 [proxy:0:1@opt10] got pmi command (from 4): get kvsname=kvs_5945_0 key=PMI_process_mapping [proxy:0:1@opt10] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [mpiexec@opt09] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1@opt10] got pmi command (from 4): barrier_in [proxy:0:1@opt10] forwarding command (cmd=barrier_in) upstream [proxy:0:0@opt09] got pmi command (from 4): init pmi_version=1 pmi_subversion=1 [proxy:0:0@opt09] PMI response: cmd=response_to_init pmi_version=1 pmi_subversion=1 rc=0 [proxy:0:0@opt09] got pmi command (from 4): get_maxes [proxy:0:0@opt09] PMI response: cmd=maxes kvsname_max=256 keylen_max=64 vallen_max=1024 [proxy:0:0@opt09] got pmi command (from 4): get_appnum [proxy:0:0@opt09] PMI response: cmd=appnum appnum=0 [proxy:0:0@opt09] got pmi command (from 4): get_my_kvsname [proxy:0:0@opt09] PMI response: cmd=my_kvsname kvsname=kvs_5945_0 [proxy:0:0@opt09] got pmi command (from 4): get_my_kvsname [proxy:0:0@opt09] PMI response: cmd=my_kvsname kvsname=kvs_5945_0 [proxy:0:0@opt09] got pmi command (from 4): get kvsname=kvs_5945_0 key=PMI_process_mapping [proxy:0:0@opt09] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,2,1)) [proxy:0:0@opt09] got pmi command (from 4): barrier_in [mpiexec@opt09] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@opt09] PMI response to fd 12 pid 4: cmd=barrier_out [mpiexec@opt09] PMI response to fd 6 pid 4: cmd=barrier_out [proxy:0:0@opt09] forwarding command (cmd=barrier_in) upstream [proxy:0:0@opt09] PMI response: cmd=barrier_out [proxy:0:1@opt10] PMI response: cmd=barrier_out [proxy:0:0@opt09] got pmi command (from 4): put [proxy:0:1@opt10] got pmi command (from 4): put kvsname=kvs_5945_0 kvsname=kvs_5945_0 key=P1-businesscard value=description#opt10$port#44966$ifname#127.0.0.1$ [proxy:0:1@opt10] we don't understand this command put; forwarding upstream [mpiexec@opt09] [pgid: 0] got PMI command: cmd=put kvsname=kvs_5945_0 key=P1-businesscard value=description#opt10$port#44966$ifname#127.0.0.1$ [mpiexec@opt09] PMI response to fd 6 pid 4: cmd=put_result rc=0 msg=success key=P0-businesscard value=description#opt09$port#59648$ifname#127.0.0.1$ [proxy:0:0@opt09] we don't understand this command put; forwarding upstream [proxy:0:1@opt10] we don't understand the response put_result; forwarding downstream [mpiexec@opt09] [pgid: 0] got PMI command: cmd=put kvsname=kvs_5945_0 key=P0-businesscard value=description#opt09$port#59648$ifname#127.0.0.1$ [mpiexec@opt09] PMI response to fd 12 pid 4: cmd=put_result rc=0 msg=success [mpiexec@opt09] [pgid: 0] got PMI command: cmd=barrier_in [proxy:0:1@opt10] got pmi command (from 4): barrier_in [proxy:0:1@opt10] forwarding command (cmd=barrier_in) upstream [proxy:0:0@opt09] we don't understand the response put_result; forwarding downstream [proxy:0:0@opt09] got pmi command (from 4): barrier_in [mpiexec@opt09] [pgid: 0] got PMI command: cmd=barrier_in [mpiexec@opt09] PMI response to fd 12 pid 4: cmd=barrier_out [mpiexec@opt09] PMI response to fd 6 pid 4: cmd=barrier_out [proxy:0:0@opt09] forwarding command (cmd=barrier_in) upstream [proxy:0:0@opt09] PMI response: cmd=barrier_out [proxy:0:1@opt10] PMI response: cmd=barrier_out Process 1 of 2 is on opt10 Process 0 of 2 is on opt09 [proxy:0:0@opt09] got pmi command (from 4): get kvsname=kvs_5945_0 key=P1-businesscard [mpiexec@opt09] [pgid: 0] got PMI command: cmd=get kvsname=kvs_5945_0 key=P1-businesscard [mpiexec@opt09] PMI response to fd 12 pid 4: cmd=get_result rc=0 msg=success value=description#opt10$port#44966$ifname#127.0.0.1$ [proxy:0:0@opt09] forwarding command (cmd=get kvsname=kvs_5945_0 key=P1-businesscard) upstream [proxy:0:0@opt09] we don't understand the response get_result; forwarding downstream [proxy:0:0@opt09] got pmi command (from 4): get kvsname=kvs_5945_0 key=P1-businesscard [mpiexec@opt09] [pgid: 0] got PMI command: cmd=get kvsname=kvs_5945_0 key=P1-businesscard [mpiexec@opt09] PMI response to fd 12 pid 4: cmd=get_result rc=0 msg=success value=description#opt10$port#44966$ifname#127.0.0.1$ [proxy:0:0@opt09] forwarding command (cmd=get kvsname=kvs_5945_0 key=P1-businesscard) upstream [proxy:0:0@opt09] we don't understand the response get_result; forwarding downstream APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)