[mpich-discuss] Problems running mpiexec on lsf
Farkash, Noam
Noam_Farkash at mentor.com
Tue May 8 12:24:02 CDT 2012
Hi,
I'm having problems running mpiexec on an lsf system. (The version I'm using is: mpich2-1.4.1)
The error I get is:
"/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy": No such file or directory
I checked that the machines (which the lsf is assigning) actually do see this file ( I manually logged into those machines and run ls&stat on that file)
In order to help debug the issue I launched mpiexec (with -verbose) on a simple program that simply does an ls&stat of the hydra_pmi_proxy file.
>cat list_file
#!/bin/csh
ls /tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
stat /tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
uname -a
>bsub -n 11 -R 'select[(type==RHEL4_64) && cpuf>45 && (mem>10000||gb32||gb64)] rusage[mem=10000]' -P bd-ckt -q normal -o junk /tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec -n 11 list_file
Results of the run follow:
host: svbr0124
host: ca2h2388
==================================================================================================
mpiexec options:
----------------
Base path: /tool/cbar/apps/in/10.1-BETA.3/arch/lib/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
MANPATH=/tool/lsf7/man:.:/usr/local/man:/usr/man
LSF_EAUTH_AUX_PASS=yes
SSH_AGENT_PID=1156
HOSTNAME=svbr0124
TECH_HOME=/tool/cbar/apps/in/10.1-BETA.3/arch/ztech
LSF_LIM_API_NTRIES=1
LSF_LOGDIR=/tmp
LSB_BATCH_JID=518497
HOST=lx-deskh448
SHELL=/tool/pandora/bin/tcsh
DESKTOP_STARTUP_ID=
PERL5LIB=/proj/verif_release_ro/p4w/114:/proj/verif_release_ro/p4_mkwa/current:/proj/verif_release_ro/siteconfig/current/lib/perl:/proj/verif_release_ro/error_reporting/25/lib/perl:/proj/verif_release_ro/cbwa_bootcore/current/lib/perl
LS_JOBPID=23044
CDC_PREW2KHOST=lx-deskh448
GTK_RC_FILES=/etc/gtk/gtkrc:/home/joe/.gtkrc-1.2-gnome2
LSB_JOBRES_CALLBACK=50786 at svbr0124
LM_PROJECT=DIV-CMD:SITE-CAL:PROJ-unknown
WINDOWID=61212406
LSB_JOB_EXECUSER=joe
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
TB_ADC_MACHINE_BASE=adcgar0
TB_MHDC_MACHINE_BASE=mhdcgar0
ANCHOR_p4_mkwa=/proj/verif_release_ro/p4_mkwa/current
SVN_SSH_SCRIPT=/proj/verif_release_ro/svn_ssh/27/svnssh
BSUB_BLOCK_EXEC_HOST=
USER=joe
GROUP=design
LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:
SBD_KRB5CCNAME_VAL=
MAGIC_SITENAME=SVDC
GLOBAL_VERIF_CVSROOT=:pserver:@mpdtxcvs.abc.com:/proj/verif/cvsroot
HOME_IN_PATH=/tool/cbar/apps/in/10.1-BETA.3/arch/bin
SSH_AUTH_SOCK=/tmp/ssh-NjSRhC1115/agent.1115
GNOME_KEYRING_SOCKET=/tmp/keyring-9SZs9i/socket
LSF_INVOKE_CMD=bsub
KDEDIR=/usr
USERNAME=joe
SESSION_MANAGER=local/lx-deskh448:/tmp/.ICE-unix/1189
LSF_EAUTH_SERVER=mbatchd at ca
LS_SUBCWD=/proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF
LSF_VERSION=20
LSB_UNIXGROUP_INT=design
MAIL=/usr/spool/mail/joe
DESKTOP_SESSION=default
ZERO_IND_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com
XREMOTETMPDIR=/home/joe
LSB_EXIT_PRE_ABORT=99
LSB_JOBEXIT_STAT=0
GDM_XSERVER_LOCATION=local
TB_BDC_MACHINE_BASE=bdcgar0
SETUP_MODULES=3.0b1
INPUTRC=/etc/inputrc
KDE_IS_PRELINKED=1
LANG=C
LSF_EAUTH_CLIENT=user
MODULEPATH=/tool/abc-env/modules:/proj/verif_release_ro:/tool/pandora64/etc/modules:/tool/pandora/etc/modules:/tool/cbar/etc/SVDC/modules:/proj/verif_release_ro/modules/current:/proj/verif/lib/modules:/usr/local/modules:/tool/cbar/etc/modules
LOADEDMODULES=cbwa_bootcore/current/modulefile:error_reporting/25/modulefile:siteconfig/current/modulefile:perl/5.8.8:tcltk/8.4.6:svn_ssh/27/modulefile:subversion/1.6.3:p4_mkwa/current/modulefile:p4w/114/modulefile:in/10.1-BETA.3
USER_PRINCIPAL_NAME=joe at ABC.COM
TCLSH=/tool/pandora64/.package/tcltk-8.4.6/bin/tclsh
GDMSESSION=default
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
KRB5CCNAME=FILE:/tmp/krb5cc_cdc41374_0QfhVk
SVN_SSH_PKG=/proj/verif_release_ro/svn_ssh/27
CAD=/proj/cad
CDC_JOINED_DC=ssvldc4.abc.com
TCLTK_HOME=/tool/pandora64/.package/tcltk-8.4.6
CADTOOLS=/tools/cad
SHLVL=3
HOME_IN=/tool/cbar/apps/in/10.1-BETA.3/arch
OSTYPE=linux
CDC_JOINED_SITE=Nowh
GNOME_DESKTOP_SESSION_ID=Default
SUBVERSION_HOME=/tool/pandora64/.package/subversion-1.6.3
CBW_RELEASED_CONFIG_ROOT=none
LSB_SUB_HOST=lx-deskh448
TB_SVDC_MACHINE_BASE=svdcgar0
LSFUSER=joe
RCSINIT=-zLT
VENDOR=intel
CBWA_USTAT_SEND=/proj/verif_release_ro/ustat_send/16
LSB_QUEUE=normal
MACHTYPE=i386
LOGNAME=joe
QTLIB=/usr/lib64/qt-3.3/lib
PRINTER=cmdlpr2
CVS_RSH=cvsssh
MGLS_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com
XLSF_UIDDIR=/tool/lsf7/lib/uid
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-EGrrnZy1IW,guid=129533580524e6f852392b004f708a24
MODULESHOME=/tool/pandora/.package/modulecmd-tcl-abc-1.03
LESSOPEN=|/usr/bin/lesspipe.sh %s
CDC_JOINED_ZONE=CN=design,OU=zones,OU=EngUnix,DC=abc,DC=com
ANCHOR_p4w=/proj/verif_release_ro/p4w/114
ACME_SVN_ALIAS=svn --config-dir /proj/verif_release_ro/svn_ssh/27/svn_config
CDC_LOCALHOST=lx-deskh448.abc.com
LSF_ENVDIR=/tool/lsf7/conf
FLEXLM_NO_CKOUT_INSTALL_LIC=1
SITECONFIG_DATAFILE=/proj/verif_release_ro/siteconfig/current/lib/perl/SiteConfig.TXT
GLOBAL_VERIF_PATH=/proj/verif
G_BROKEN_FILENAMES=1
P4CONFIG=P4CONFIG
CDC_JOINED_DOMAIN=abc.com
XAUTHORITY=/tmp/.gdmV8MVBW
COLORTERM=gnome-terminal
_=/tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: svbr0124 (10 cores)
Exec list: list_file (10 processes);
[2] proxy: ca2h2388 (1 cores)
Exec list: list_file (1 processes);
==================================================================================================
[mpiexec at svbr0124] Timeout set to -1 (-1 means infinite)
[mpiexec at svbr0124] Got a control port string of svbr0124:50789
Proxy launch args: /tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy --control-port svbr0124:50789 --debug --rmk lsf --launcher lsf --demux poll --pgid 0 --retries 10 --proxy-id
[mpiexec at svbr0124] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 0:
--version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname svbr0124 --global-core-map 0,10,1 --filler-process-map 0,10,1 --global-process-count 11 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_23053_0 --pmi-process-mapping (vector,(0,1,10),(1,1,1)) --ckpoint-num -1 --global-inherited-env 106 'MANPATH=/tool/lsf7/man:.:/usr/local/man:/usr/man' 'LSF_EAUTH_AUX_PASS=yes' 'SSH_AGENT_PID=1156' 'HOSTNAME=svbr0124' 'TECH_HOME=/tool/cbar/apps/in/10.1-BETA.3/arch/ztech' 'LSF_LIM_API_NTRIES=1' 'LSF_LOGDIR=/tmp' 'LSB_BATCH_JID=518497' 'HOST=lx-deskh448' 'SHELL=/tool/pandora/bin/tcsh' 'DESKTOP_STARTUP_ID=' 'PERL5LIB=/proj/verif_release_ro/p4w/114:/proj/verif_release_ro/p4_mkwa/current:/proj/verif_release_ro/siteconfig/current/lib/perl:/proj/verif_release_ro/error_reporting/25/lib/perl:/proj/verif_release_ro/cbwa_bootcore/current/lib/perl' 'LS_JOBPID=23044' 'CDC_PREW2KHOST=lx-deskh448' 'GTK_RC_FILES=/etc/gtk/gtkrc:/home/joe/.gtkrc-1.2-gnome2' 'LSB_JOBRES_CALLBACK=50786 at svbr0124' 'LM_PROJECT=DIV-CMD:SITE-CAL:PROJ-unknown' 'WINDOWID=61212406' 'LSB_JOB_EXECUSER=joe' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'TB_ADC_MACHINE_BASE=adcgar0' 'TB_MHDC_MACHINE_BASE=mhdcgar0' 'ANCHOR_p4_mkwa=/proj/verif_release_ro/p4_mkwa/current' 'SVN_SSH_SCRIPT=/proj/verif_release_ro/svn_ssh/27/svnssh' 'BSUB_BLOCK_EXEC_HOST=' 'USER=joe' 'GROUP=design' 'LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:' 'SBD_KRB5CCNAME_VAL=' 'MAGIC_SITENAME=SVDC' 'GLOBAL_VERIF_CVSROOT=:pserver:@mpdtxcvs.abc.com:/proj/verif/cvsroot' 'HOME_IN_PATH=/tool/cbar/apps/in/10.1-BETA.3/arch/bin' 'SSH_AUTH_SOCK=/tmp/ssh-NjSRhC1115/agent.1115' 'GNOME_KEYRING_SOCKET=/tmp/keyring-9SZs9i/socket' 'LSF_INVOKE_CMD=bsub' 'KDEDIR=/usr' 'USERNAME=joe' 'SESSION_MANAGER=local/lx-deskh448:/tmp/.ICE-unix/1189' 'LSF_EAUTH_SERVER=mbatchd at ca' 'LS_SUBCWD=/proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF' 'LSF_VERSION=20' 'LSB_UNIXGROUP_INT=design' 'MAIL=/usr/spool/mail/joe' 'DESKTOP_SESSION=default' 'ZERO_IND_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com' 'XREMOTETMPDIR=/home/joe' 'LSB_EXIT_PRE_ABORT=99' 'LSB_JOBEXIT_STAT=0' 'GDM_XSERVER_LOCATION=local' 'TB_BDC_MACHINE_BASE=bdcgar0' 'SETUP_MODULES=3.0b1' 'INPUTRC=/etc/inputrc' 'KDE_IS_PRELINKED=1' 'LANG=C' 'LSF_EAUTH_CLIENT=user' 'MODULEPATH=/tool/abc-env/modules:/proj/verif_release_ro:/tool/pandora64/etc/modules:/tool/pandora/etc/modules:/tool/cbar/etc/SVDC/modules:/proj/verif_release_ro/modules/current:/proj/verif/lib/modules:/usr/local/modules:/tool/cbar/etc/modules' 'LOADEDMODULES=cbwa_bootcore/current/modulefile:error_reporting/25/modulefile:siteconfig/current/modulefile:perl/5.8.8:tcltk/8.4.6:svn_ssh/27/modulefile:subversion/1.6.3:p4_mkwa/current/modulefile:p4w/114/modulefile:in/10.1-BETA.3' 'USER_PRINCIPAL_NAME=joe at ABC.COM' 'TCLSH=/tool/pandora64/.package/tcltk-8.4.6/bin/tclsh' 'GDMSESSION=default' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'KRB5CCNAME=FILE:/tmp/krb5cc_cdc41374_0QfhVk' 'SVN_SSH_PKG=/proj/verif_release_ro/svn_ssh/27' 'CAD=/proj/cad' 'CDC_JOINED_DC=ssvldc4.abc.com' 'TCLTK_HOME=/tool/pandora64/.package/tcltk-8.4.6' 'CADTOOLS=/tools/cad' 'SHLVL=3' 'HOME_IN=/tool/cbar/apps/in/10.1-BETA.3/arch' 'OSTYPE=linux' 'CDC_JOINED_SITE=Nowh' 'GNOME_DESKTOP_SESSION_ID=Default' 'SUBVERSION_HOME=/tool/pandora64/.package/subversion-1.6.3' 'CBW_RELEASED_CONFIG_ROOT=none' 'LSB_SUB_HOST=lx-deskh448' 'TB_SVDC_MACHINE_BASE=svdcgar0' 'LSFUSER=joe' 'RCSINIT=-zLT' 'VENDOR=intel' 'CBWA_USTAT_SEND=/proj/verif_release_ro/ustat_send/16' 'LSB_QUEUE=normal' 'MACHTYPE=i386' 'LOGNAME=joe' 'QTLIB=/usr/lib64/qt-3.3/lib' 'PRINTER=cmdlpr2' 'CVS_RSH=cvsssh' 'MGLS_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com' 'XLSF_UIDDIR=/tool/lsf7/lib/uid' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-EGrrnZy1IW,guid=129533580524e6f852392b004f708a24' 'MODULESHOME=/tool/pandora/.package/modulecmd-tcl-abc-1.03' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'CDC_JOINED_ZONE=CN=design,OU=zones,OU=EngUnix,DC=abc,DC=com' 'ANCHOR_p4w=/proj/verif_release_ro/p4w/114' 'ACME_SVN_ALIAS=svn --config-dir /proj/verif_release_ro/svn_ssh/27/svn_config' 'CDC_LOCALHOST=lx-deskh448.abc.com' 'LSF_ENVDIR=/tool/lsf7/conf' 'FLEXLM_NO_CKOUT_INSTALL_LIC=1' 'SITECONFIG_DATAFILE=/proj/verif_release_ro/siteconfig/current/lib/perl/SiteConfig.TXT' 'GLOBAL_VERIF_PATH=/proj/verif' 'G_BROKEN_FILENAMES=1' 'P4CONFIG=P4CONFIG' 'CDC_JOINED_DOMAIN=abc.com' 'XAUTHORITY=/tmp/.gdmV8MVBW' 'COLORTERM=gnome-terminal' '_=/tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 10 --exec --exec-appnum 0 --exec-proc-count 10 --exec-local-env 0 --exec-wdir /proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF --exec-args 1 list_file
[mpiexec at svbr0124] PMI FD: (null); PMI PORT: (null); PMI ID/RANK: -1
Arguments being passed to proxy 1:
--version 1.4.1 --iface-ip-env-name MPICH_INTERFACE_HOSTNAME --hostname ca2h2388 --global-core-map 10,1,0 --filler-process-map 10,1,0 --global-process-count 11 --auto-cleanup 1 --pmi-rank -1 --pmi-kvsname kvs_23053_0 --pmi-process-mapping (vector,(0,1,10),(1,1,1)) --ckpoint-num -1 --global-inherited-env 106 'MANPATH=/tool/lsf7/man:.:/usr/local/man:/usr/man' 'LSF_EAUTH_AUX_PASS=yes' 'SSH_AGENT_PID=1156' 'HOSTNAME=svbr0124' 'TECH_HOME=/tool/cbar/apps/in/10.1-BETA.3/arch/ztech' 'LSF_LIM_API_NTRIES=1' 'LSF_LOGDIR=/tmp' 'LSB_BATCH_JID=518497' 'HOST=lx-deskh448' 'SHELL=/tool/pandora/bin/tcsh' 'DESKTOP_STARTUP_ID=' 'PERL5LIB=/proj/verif_release_ro/p4w/114:/proj/verif_release_ro/p4_mkwa/current:/proj/verif_release_ro/siteconfig/current/lib/perl:/proj/verif_release_ro/error_reporting/25/lib/perl:/proj/verif_release_ro/cbwa_bootcore/current/lib/perl' 'LS_JOBPID=23044' 'CDC_PREW2KHOST=lx-deskh448' 'GTK_RC_FILES=/etc/gtk/gtkrc:/home/joe/.gtkrc-1.2-gnome2' 'LSB_JOBRES_CALLBACK=50786 at svbr0124' 'LM_PROJECT=DIV-CMD:SITE-CAL:PROJ-unknown' 'WINDOWID=61212406' 'LSB_JOB_EXECUSER=joe' 'QTDIR=/usr/lib64/qt-3.3' 'QTINC=/usr/lib64/qt-3.3/include' 'TB_ADC_MACHINE_BASE=adcgar0' 'TB_MHDC_MACHINE_BASE=mhdcgar0' 'ANCHOR_p4_mkwa=/proj/verif_release_ro/p4_mkwa/current' 'SVN_SSH_SCRIPT=/proj/verif_release_ro/svn_ssh/27/svnssh' 'BSUB_BLOCK_EXEC_HOST=' 'USER=joe' 'GROUP=design' 'LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:' 'SBD_KRB5CCNAME_VAL=' 'MAGIC_SITENAME=SVDC' 'GLOBAL_VERIF_CVSROOT=:pserver:@mpdtxcvs.abc.com:/proj/verif/cvsroot' 'HOME_IN_PATH=/tool/cbar/apps/in/10.1-BETA.3/arch/bin' 'SSH_AUTH_SOCK=/tmp/ssh-NjSRhC1115/agent.1115' 'GNOME_KEYRING_SOCKET=/tmp/keyring-9SZs9i/socket' 'LSF_INVOKE_CMD=bsub' 'KDEDIR=/usr' 'USERNAME=joe' 'SESSION_MANAGER=local/lx-deskh448:/tmp/.ICE-unix/1189' 'LSF_EAUTH_SERVER=mbatchd at ca' 'LS_SUBCWD=/proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF' 'LSF_VERSION=20' 'LSB_UNIXGROUP_INT=design' 'MAIL=/usr/spool/mail/joe' 'DESKTOP_SESSION=default' 'ZERO_IND_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com' 'XREMOTETMPDIR=/home/joe' 'LSB_EXIT_PRE_ABORT=99' 'LSB_JOBEXIT_STAT=0' 'GDM_XSERVER_LOCATION=local' 'TB_BDC_MACHINE_BASE=bdcgar0' 'SETUP_MODULES=3.0b1' 'INPUTRC=/etc/inputrc' 'KDE_IS_PRELINKED=1' 'LANG=C' 'LSF_EAUTH_CLIENT=user' 'MODULEPATH=/tool/abc-env/modules:/proj/verif_release_ro:/tool/pandora64/etc/modules:/tool/pandora/etc/modules:/tool/cbar/etc/SVDC/modules:/proj/verif_release_ro/modules/current:/proj/verif/lib/modules:/usr/local/modules:/tool/cbar/etc/modules' 'LOADEDMODULES=cbwa_bootcore/current/modulefile:error_reporting/25/modulefile:siteconfig/current/modulefile:perl/5.8.8:tcltk/8.4.6:svn_ssh/27/modulefile:subversion/1.6.3:p4_mkwa/current/modulefile:p4w/114/modulefile:in/10.1-BETA.3' 'USER_PRINCIPAL_NAME=joe at ABC.COM' 'TCLSH=/tool/pandora64/.package/tcltk-8.4.6/bin/tclsh' 'GDMSESSION=default' 'SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass' 'KRB5CCNAME=FILE:/tmp/krb5cc_cdc41374_0QfhVk' 'SVN_SSH_PKG=/proj/verif_release_ro/svn_ssh/27' 'CAD=/proj/cad' 'CDC_JOINED_DC=ssvldc4.abc.com' 'TCLTK_HOME=/tool/pandora64/.package/tcltk-8.4.6' 'CADTOOLS=/tools/cad' 'SHLVL=3' 'HOME_IN=/tool/cbar/apps/in/10.1-BETA.3/arch' 'OSTYPE=linux' 'CDC_JOINED_SITE=Sunnyvale' 'GNOME_DESKTOP_SESSION_ID=Default' 'SUBVERSION_HOME=/tool/pandora64/.package/subversion-1.6.3' 'CBW_RELEASED_CONFIG_ROOT=none' 'LSB_SUB_HOST=lx-deskh448' 'TB_SVDC_MACHINE_BASE=svdcgar0' 'LSFUSER=joe' 'RCSINIT=-zLT' 'VENDOR=intel' 'CBWA_USTAT_SEND=/proj/verif_release_ro/ustat_send/16' 'LSB_QUEUE=normal' 'MACHTYPE=i386' 'LOGNAME=joe' 'QTLIB=/usr/lib64/qt-3.3/lib' 'PRINTER=cmdlpr2' 'CVS_RSH=cvsssh' 'MGLS_LICENSE_FILE=27003 at cremer.abc.com:27003 at licsrv05.abc.com,27003 at licsrv05-f.abc.com' 'XLSF_UIDDIR=/tool/lsf7/lib/uid' 'DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-EGrrnZy1IW,guid=129533580524e6f852392b004f708a24' 'MODULESHOME=/tool/pandora/.package/modulecmd-tcl-abc-1.03' 'LESSOPEN=|/usr/bin/lesspipe.sh %s' 'CDC_JOINED_ZONE=CN=design,OU=zones,OU=EngUnix,DC=abc,DC=com' 'ANCHOR_p4w=/proj/verif_release_ro/p4w/114' 'ACME_SVN_ALIAS=svn --config-dir /proj/verif_release_ro/svn_ssh/27/svn_config' 'CDC_LOCALHOST=lx-deskh448.abc.com' 'LSF_ENVDIR=/tool/lsf7/conf' 'FLEXLM_NO_CKOUT_INSTALL_LIC=1' 'SITECONFIG_DATAFILE=/proj/verif_release_ro/siteconfig/current/lib/perl/SiteConfig.TXT' 'GLOBAL_VERIF_PATH=/proj/verif' 'G_BROKEN_FILENAMES=1' 'P4CONFIG=P4CONFIG' 'CDC_JOINED_DOMAIN=abc.com' 'XAUTHORITY=/tmp/.gdmV8MVBW' 'COLORTERM=gnome-terminal' '_=/tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF --exec-args 1 list_file
[mpiexec at svbr0124] Launch arguments: /tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy --control-port svbr0124:50789 --debug --rmk lsf --launcher lsf --demux poll --pgid 0 --retries 10 --proxy-id 0
[mpiexec at svbr0124] Launch arguments: /tool/lsf7/bin/blaunch -n ca2h2388 "/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy" --control-port svbr0124:50789 --debug --rmk lsf --launcher lsf --demux poll --pgid 0 --retries 10 --proxy-id 1
"/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy": No such file or directory
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
File: `/tool/cbar/apps/in/10.1-BETA.3/arch/lib/hydra_pmi_proxy'
Size: 846967 Blocks: 1664 IO Block: 32768 regular file
Device: 1dh/29d Inode: 52716957 Links: 1
Access: (0755/-rwxr-xr-x) Uid: (15556/ cbaradm) Gid: ( 6074/ cbar)
Access: 2012-05-02 17:48:27.000000000 -0700
Modify: 2012-03-06 14:09:20.000000000 -0800
Change: 2012-04-20 07:35:29.000000000 -0700
Linux svbr0124 2.6.9-100.ELlargesmp #1 SMP Tue Feb 1 12:09:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
------------------------------------------------------------
Sender: LSF System <lsfadmin at svbr0124>
Subject: Job 518497: </tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec -n 11 -verbose list_file> Exited
Job </tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec -n 11 -verbose list_file> was submitted from host <lx-deskh448> by user <joe>.
Job was executed on host(s) <10*svbr0124>, in queue <normal>, as user <joe>.
<1*ca2h2388>
</home/joe> was used as the home directory.
</proj/bd_verif_sv_scratch2/users/joe/test_case_for_xyz/AREXAL0FRF> was used as the working directory.
Started at Thu May 3 10:35:15 2012
Results reported at Fri May 4 11:55:14 2012
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
/tool/cbar/apps/in/10.1-BETA.3/arch/lib/mpiexec -n 11 -verbose list_file
------------------------------------------------------------
TERM_OWNER: job killed by owner.
Exited with signal termination: Killed.
Resource usage summary:
CPU time : 0.15 sec.
Max Memory : 2 MB
Max Swap : 19 MB
Max Processes : 1
Max Threads : 1
The output (if any) is above this job summary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20120508/9fe84cc3/attachment-0001.htm>
More information about the mpich-discuss
mailing list