[mpich-discuss] SGE & Hydra Problem
Pavan Balaji
balaji at mcs.anl.gov
Tue Sep 14 13:06:45 CDT 2010
Hello,
Please keep mpich-discuss cc'ed.
The below output seems incomplete. Did the launch hang, and had to be
killed?
Can you try the below command?
% /installadmin/sge/bin/lx24-amd64/qrsh -inherit -V b56
/installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
It should launch and error out, but not hang.
-- Pavan
On 09/14/2010 04:26 AM, Ursula Winkler wrote:
> Pavan Balaji schrieb:
>>
>>
>> Can you run this by passing the -verbose option to mpiexec? It'll give
>> some more output to help us debug it.
>>
>
> There should be processes with 8 nodes (b73 is the job master), but there
> aren't any processes on the slave nodes, so there is also no communication:
>
> mpiexec -verbose ./cpitest.x :
>
> mpiexec options:
> ----------------
> Base path: /installadmin/mpich2/test/intel/bin/
> Bootstrap server: (null)
> Debug level: 1
> Enable X: -1
>
> Global environment:
> -------------------
> REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT
>
> MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man
> CONSOLE=/dev/console
> SELINUX_INIT=YES
>
> INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses
> HOST=b00
> TERM=xterm
> HISTSIZE=1000
> SSH_CLIENT=143.50.128.178 34329 22
> SSH_TTY=/dev/pts/1
> GROUP=edvz
>
> LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64
> LS_COLORS=no
> INIT_VERSION=sysvinit-2.86
> HOSTTYPE=x86_64-linux
> AUTOBOOT=YES
> MAIL=/var/spool/mail/winkl
> runlevel=3
> RUNLEVEL=3
> INPUTRC=/etc/inputrc
> PWD=/usr/people/edvz/winkl/MPI-Test
> SGE_ACCOUNT=sge
> LANG=en_US.UTF-8
> previous=N
> PREVLEVEL=N
> REQNAME=test_nodes.b2
> SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> MPI=/installadmin/mpich2/test/intel
> SHLVL=2
> SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test
> OSTYPE=linux
> BOOT_IMAGE=2.6.18-194.11.3
> MPIHOME=/installadmin/mpich2/test/intel
> VENDOR=unknown
> MACHTYPE=x86_64
> CVS_RSH=ssh
> SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22
> LESSOPEN=|/usr/bin/lesspipe.sh %s
> G_BROKEN_FILENAMES=1
> _=/installadmin/mpich2/test/intel/bin/mpiexec
>
>
> Proxy information:
> *********************
> Proxy ID: 1
> -----------------
> Proxy name: b73
> Process count: 2
> Start PID: 0
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 2
> -----------------
> Proxy name: b56
> Process count: 2
> Start PID: 2
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 3
> -----------------
> Proxy name: b58
> Process count: 2
> Start PID: 4
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 4
> -----------------
> Proxy name: b54
> Process count: 2
> Start PID: 6
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 5
> -----------------
> Proxy name: b74
> Process count: 2
> Start PID: 8
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 6
> -----------------
> Proxy name: b75
> Process count: 2
> Start PID: 10
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 7
> -----------------
> Proxy name: b65
> Process count: 2
> Start PID: 12
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
> Proxy ID: 8
> -----------------
> Proxy name: b55
> Process count: 2
> Start PID: 14
>
> Proxy exec list:
> ....................
> Exec: ./cpitest.x; Process count: 2
>
> ==================================================================================================
>
> [mpiexec at b73] Timeout set to -1 (-1 means infinite)
> [mpiexec at b73] Got a control port string of b73:52298
>
> Proxy launch args: /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 0:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b73
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 0 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 1:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b56
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 2 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 2:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b58
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 4 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 3:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b54
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 6 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 4:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b74
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 8 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 5:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b75
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 10 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 6:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b65
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 12 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
> Arguments being passed to proxy 7:
> --version 1.3b1 --interface-env-name MPICH_INTERFACE_NAME --hostname b55
> --global-core-count 16 --global-process-count 16 --auto-cleanup 1
> --pmi-rank -1 --pmi-kvsname kvs_7149_0 --pmi-process-mapping
> (vector,(0,8,2)) --global-inherited-env 40
> 'REMOTEHOST=ZID178.KFUNIGRAZ.AC.AT'
> 'MANPATH=/installadmin/sge/man:/software/mpich2/test/intel/share/man:/software/intel/intel_fce_111/man:/software/intel/intel_cce_111/man:/installadmin/sge/man:/usr/share/man/en:/usr/share/man:/usr/local/share/man'
> 'CONSOLE=/dev/console' 'SELINUX_INIT=YES'
> 'INTEL_LICENSE_FILE=/software/intel/intel_fce_111/licenses:/opt/intel/licenses:/usr/people/edvz/winkl/intel/licenses:/software/intel/intel_cce_111/licenses:/software/intel/licenses:/usr/people/edvz/winkl/intel/licenses'
> 'HOST=b00' 'TERM=xterm' 'HISTSIZE=1000' 'SSH_CLIENT=143.50.128.178 34329
> 22' 'SSH_TTY=/dev/pts/1' 'GROUP=edvz'
> 'LD_LIBRARY_PATH=/installadmin/mpich2/test/intel/lib:/software/intel/intel_fce_111/lib/intel64:/software/intel/intel_cce_111/lib/intel64'
> 'LS_COLORS=no' 'INIT_VERSION=sysvinit-2.86' 'HOSTTYPE=x86_64-linux'
> 'AUTOBOOT=YES' 'MAIL=/var/spool/mail/winkl' 'runlevel=3' 'RUNLEVEL=3'
> 'INPUTRC=/etc/inputrc' 'PWD=/usr/people/edvz/winkl/MPI-Test'
> 'SGE_ACCOUNT=sge' 'LANG=en_US.UTF-8' 'previous=N' 'PREVLEVEL=N'
> 'REQNAME=test_nodes.b2'
> 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass'
> 'MPI=/installadmin/mpich2/test/intel' 'SHLVL=2'
> 'SGE_CWD_PATH=/usr/people/edvz/winkl/MPI-Test' 'OSTYPE=linux'
> 'BOOT_IMAGE=2.6.18-194.11.3' 'MPIHOME=/installadmin/mpich2/test/intel'
> 'VENDOR=unknown' 'MACHTYPE=x86_64' 'CVS_RSH=ssh'
> 'SSH_CONNECTION=143.50.128.178 34329 143.50.10.40 22'
> 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'G_BROKEN_FILENAMES=1'
> '_=/installadmin/mpich2/test/intel/bin/mpiexec' --global-user-env 0
> --global-system-env 0 --start-pid 14 --proxy-core-count 2 --exec
> --exec-appnum 0 --exec-proc-count 2 --exec-local-env 0 --exec-wdir
> /usr/people/edvz/winkl/MPI-Test --exec-args 1 ./cpitest.x
>
> [mpiexec at b73] Launch arguments:
> /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy --control-port
> b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1 --proxy-id 0
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b56 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 1
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b58 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 2
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b54 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 3
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b74 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 4
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b75 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 5
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b65 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 6
> [mpiexec at b73] Launch arguments: /installadmin/sge/bin/lx24-amd64/qrsh
> -inherit -V b55 /installadmin/mpich2/test/intel/bin/hydra_pmi_proxy
> --control-port b73:52298 --debug --demux poll --pgid 0 --enable-stdin 1
> --proxy-id 7
> [proxy:0:0 at b73] got pmi command (from 9): init
> pmi_version=1 pmi_subversion=1
> [proxy:0:0 at b73] PMI response: cmd=response_to_init pmi_version=1
> pmi_subversion=1 rc=0
> [proxy:0:0 at b73] got pmi command (from 6): init
> pmi_version=1 pmi_subversion=1
> [proxy:0:0 at b73] PMI response: cmd=response_to_init pmi_version=1
> pmi_subversion=1 rc=0
> [proxy:0:0 at b73] got pmi command (from 6): get_maxes
>
> [proxy:0:0 at b73] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
> vallen_max=1024
> [proxy:0:0 at b73] got pmi command (from 9): get_maxes
>
> [proxy:0:0 at b73] PMI response: cmd=maxes kvsname_max=256 keylen_max=64
> vallen_max=1024
> [proxy:0:0 at b73] got pmi command (from 6): get_appnum
>
> [proxy:0:0 at b73] PMI response: cmd=appnum appnum=0
> [proxy:0:0 at b73] got pmi command (from 6): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 9): get_appnum
>
> [proxy:0:0 at b73] PMI response: cmd=appnum appnum=0
> [proxy:0:0 at b73] got pmi command (from 6): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 9): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 6): get
> kvsname=kvs_7149_0 key=PMI_process_mapping
> [proxy:0:0 at b73] PMI response: cmd=get_result rc=0 msg=success
> value=(vector,(0,8,2))
> [proxy:0:0 at b73] got pmi command (from 9): get_my_kvsname
>
> [proxy:0:0 at b73] PMI response: cmd=my_kvsname kvsname=kvs_7149_0
> [proxy:0:0 at b73] got pmi command (from 6): put
> kvsname=kvs_7149_0 key=sharedFilename[0] value=/dev/shm/mpich_shar_tmpBKKMfK
> [proxy:0:0 at b73] we don't understand this command put; forwarding upstream
> [mpiexec at b73] [pgid: 0] got PMI command: cmd=put kvsname=kvs_7149_0
> key=sharedFilename[0] value=/dev/shm/mpich_shar_tmpBKKMfK
> [mpiexec at b73] PMI response to fd 6 pid 6: cmd=put_result rc=0 msg=success
> [proxy:0:0 at b73] we don't understand the response put_result; forwarding
> downstream
> [proxy:0:0 at b73] got pmi command (from 6): barrier_in
>
> [proxy:0:0 at b73] got pmi command (from 9): get
> kvsname=kvs_7149_0 key=PMI_process_mapping
> [proxy:0:0 at b73] PMI response: cmd=get_result rc=0 msg=success
> value=(vector,(0,8,2))
> [proxy:0:0 at b73] got pmi command (from 9): barrier_in
>
> [mpiexec at b73] [pgid: 0] got PMI command: cmd=barrier_in
> [proxy:0:0 at b73] forwarding command (cmd=barrier_in) upstream
>
>
>> Hydra will work out-of-the-box with MVAPICH2 (or any other derivative of
>> MPICH2). I believe the latest version of MVAPICH-1 also supports the PMI
>> interface, and hence Hydra and all other MPICH2 process managers.
>>
>>
>>
>
> Thank you for the info.
>
> Ursula
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
More information about the mpich-discuss
mailing list