[mpich-discuss] SGE & Hydra Problem

Pavan Balaji balaji at mcs.anl.gov
Tue Sep 14 13:06:45 CDT 2010


Hello,

Please keep mpich-discuss cc'ed.

The below output seems incomplete. Did the launch hang, and had to be 
killed?

Can you try the below command?

% /installadmin/sge/bin/lx24-amd64/qrsh -inherit -V b56 
/installadmin/mpich2/test/intel/bin/hydra_pmi_proxy

It should launch and error out, but not hang.

  -- Pavan

On 09/14/2010 04:26 AM, Ursula Winkler wrote:
> Pavan Balaji schrieb:
>>
>>
>> Can you run this by passing the -verbose option to mpiexec? It'll give
>> some more output to help us debug it.
>>
>
> There should be processes with 8 nodes (b73 is the job master), but there
> aren't any processes on the slave nodes, so there is also no communication:
>
> mpiexec -verbose ./cpitest.x   :
>
> mpiexec options:
> ----------------
>    Base path: /installadmin/mpich2/test/intel/bin/
>    Bootstrap server: (null)
>    Debug level: 1
>    Enable X: -1
>
>    Global environment:
>    -------------------
>      REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT
>
> MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man
>      CONSOLE=/dev/console
>      SELINUX_INIT=YES
>
> INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses
>      HOST=b00
>      TERM=xterm
>      HISTSIZE=1000
>      SSH_CLIENT=143.50.128.178 34329 22
>      SSH_TTY=/dev/pts/1
>      GROUP=edvz
>
> LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64
>      LS_COLORS=no
>      INIT_VERSION=sysvinit-2.86
>      HOSTTYPE=x86_64-linux
>      AUTOBOOT=YES
>      MAIL=/var/spool/mail/winkl
>      runlevel=3
>      RUNLEVEL=3
>      INPUTRC=/etc/inputrc
>      PWD=/usr/people/edvz/winkl/MPI-Test
>      SGE_ACCOUNT=sge
>      LANG=en_US.UTF-8
>      previous=N
>      PREVLEVEL=N
>      REQNAME=test_nodes.b2
>      SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
>      MPI=/installadmin/mpich2/test/intel
>      SHLVL=2
>      SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test
>      OSTYPE=linux
>      BOOT_IMAGE=2.6.18-194.11.3
>      MPIHOME=/installadmin/mpich2/test/intel
>      VENDOR=unknown
>      MACHTYPE=x86_64
>      CVS_RSH=ssh
>      SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22
>      LESSOPEN=|/usr/bin/lesspipe.sh %s
>      G_BROKEN_FILENAMES=1
>      _=/installadmin/mpich2/test/intel/bin/mpiexec
>
>
>      Proxy information:
>      *********************
>        Proxy ID:  1
>        -----------------
>          Proxy name: b73
>          Process count: 2
>          Start PID: 0
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  2
>        -----------------
>          Proxy name: b56
>          Process count: 2
>          Start PID: 2
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  3
>        -----------------
>          Proxy name: b58
>          Process count: 2
>          Start PID: 4
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  4
>        -----------------
>          Proxy name: b54
>          Process count: 2
>          Start PID: 6
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  5
>        -----------------
>          Proxy name: b74
>          Process count: 2
>          Start PID: 8
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  6
>        -----------------
>          Proxy name: b75
>          Process count: 2
>          Start PID: 10
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  7
>        -----------------
>          Proxy name: b65
>          Process count: 2
>          Start PID: 12
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>        Proxy ID:  8
>        -----------------
>          Proxy name: b55
>          Process count: 2
>          Start PID: 14
>
>          Proxy exec list:
>          ....................
>            Exec: ./cpitest.x; Process count: 2
>
> ==================================================================================================
>
> [mpiexec at b73] Timeout set to -1 (-1 means infinite)
> [mpiexec at b73] Got a control port string of b73:52298
>
> Proxy launch args: /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 0:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b73
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 0 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 1:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b56
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 2 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 2:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b58
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 4 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 3:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b54
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 6 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 4:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b74
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 8 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 5:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b75
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 10 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 6:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b65
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 12 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 7:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b55
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 14 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] Launch arguments:
> /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy --control-port
> b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1 --proxy-id 0
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b56 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 1
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b58 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 2
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b54 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 3
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b74 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 4
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b75 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 5
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b65 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 6
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b55 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 7
> [proxy:0:0 at b73] got pmi command (from 9): init
> pmi_version=1 pmi_subversion=1
> [proxy:0:0 at b73] PMI response: cmd=response_to_init pmi_version=1
> pmi_subversion=1 rc=0
> [proxy:0:0 at b73] got pmi command (from 6): init
> pmi_version=1 pmi_subversion=1
> [proxy:0:0 at b73] PMI response: cmd=response_to_init pmi_version=1
> pmi_subversion=1 rc=0
> [proxy:0:0 at b73] got pmi command (from 6): get_maxes
>
> [proxy:0:0 at b73] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
> vallen_max=1024
> [proxy:0:0 at b73] got pmi command (from 9): get_maxes
>
> [proxy:0:0 at b73] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
> vallen_max=1024
> [proxy:0:0 at b73] got pmi command (from 6): get_appnum
>
> [proxy:0:0 at b73] PMI response: cmd=appnum appnum=0
> [proxy:0:0 at b73] got pmi command (from 6): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 9): get_appnum
>
> [proxy:0:0 at b73] PMI response: cmd=appnum appnum=0
> [proxy:0:0 at b73] got pmi command (from 6): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 9): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 6): get
> kvsname=kvs_7149_0 key=PMI_process_mapping
> [proxy:0:0 at b73] PMI response: cmd=get_result rc=0 msg=success
> value=(vector,(0,8,2))
> [proxy:0:0 at b73] got pmi command (from 9): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 6): put
> kvsname=kvs_7149_0 key=sharedFilename[0] value=/dev/shm/mpich_shar_tmpBKKMfK
> [proxy:0:0 at b73] we don't understand this command put; forwarding upstream
> [mpiexec at b73] [pgid: 0] got PMI command: cmd=put kvsname=kvs_7149_0
> key=sharedFilename[0] value=/dev/shm/mpich_shar_tmpBKKMfK
> [mpiexec at b73] PMI response to fd 6 pid 6: cmd=put_result rc=0 msg=success
> [proxy:0:0 at b73] we don't understand the response put_result; forwarding
> downstream
> [proxy:0:0 at b73] got pmi command (from 6): barrier_in
>
> [proxy:0:0 at b73] got pmi command (from 9): get
> kvsname=kvs_7149_0 key=PMI_process_mapping
> [proxy:0:0 at b73] PMI response: cmd=get_result rc=0 msg=success
> value=(vector,(0,8,2))
> [proxy:0:0 at b73] got pmi command (from 9): barrier_in
>
> [mpiexec at b73] [pgid: 0] got PMI command: cmd=barrier_in
> [proxy:0:0 at b73] forwarding command (cmd=barrier_in) upstream
>
>
>> Hydra will work out-of-the-box with MVAPICH2 (or any other derivative of
>> MPICH2). I believe the latest version of MVAPICH-1 also supports the PMI
>> interface, and hence Hydra and all other MPICH2 process managers.
>>
>>
>>
>
> Thank you for the info.
>
> Ursula
>

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji


More information about the mpich-discuss mailing list