Dear all, <div><br></div><div>I compiled our model and created the ccsm.exe file on a linux cluster, but after I submitted a job to run it with mpirun.lsf, it failed.</div><div><div style="font-family:Tahoma;font-size:13px">
I googled this error message and found a lot of threads online. It seems to be related to how to set up the use of process manager. </div><div style="font-family:Tahoma;font-size:13px"><br></div><div><font face="Tahoma">The MPICH2 is configured with all process managers available by the administrator. The log file talks about "Hydra", but in the mpich2_wrapper file it tries to use MPD. I am not sure if this inconsistency is the cause. Any thoughts and suggestion will be appreciated.</font></div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Thanks,</div><div style="font-family:Tahoma;font-size:13px">Rui</div><div style="font-family:Tahoma;font-size:13px"> </div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Here is the log file:</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">
"[mpiexec@cn60] match_arg (./utils/args/args.c:122): unrecognized argument a</div><div style="font-family:Tahoma;font-size:13px">[mpiexec@cn60] HYDU_parse_array (./utils/args/args.c:140): argument matching returned error</div>
<div style="font-family:Tahoma;font-size:13px">[mpiexec@cn60] parse_args (./ui/mpich/utils.c:1387): error parsing input array</div><div style="font-family:Tahoma;font-size:13px">[mpiexec@cn60] HYD_uii_mpx_get_parameters (./ui/mpich/utils.c:1475): error parsing config args</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Usage: ./mpiexec [global opts] [exec1 local opts] : [exec2 local opts] : ...</div><div style="font-family:Tahoma;font-size:13px">
<br></div><div style="font-family:Tahoma;font-size:13px">Global options (passed to all executables):</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Global environment options:</div>
<div style="font-family:Tahoma;font-size:13px"> -genv {name} {value} environment variable name and value</div><div style="font-family:Tahoma;font-size:13px"> -genvlist {env1,env2,...} environment variable list to pass</div>
<div style="font-family:Tahoma;font-size:13px"> -genvnone do not pass any environment variables</div><div style="font-family:Tahoma;font-size:13px"> -genvall pass all environment variables not managed</div>
<div style="font-family:Tahoma;font-size:13px"> by the launcher (default)</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">
Other global options:</div><div style="font-family:Tahoma;font-size:13px"> -f {name} file containing the host names</div><div style="font-family:Tahoma;font-size:13px"> -hosts {host list} comma separated host list</div>
<div style="font-family:Tahoma;font-size:13px"> -wdir {dirname} working directory to use</div><div style="font-family:Tahoma;font-size:13px"> -configfile {name} config file containing MPMD launch options</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Local options (passed to individual executables):</div><div style="font-family:Tahoma;font-size:13px">
<br></div><div style="font-family:Tahoma;font-size:13px"> Local environment options:</div><div style="font-family:Tahoma;font-size:13px"> -env {name} {value} environment variable name and value</div><div style="font-family:Tahoma;font-size:13px">
-envlist {env1,env2,...} environment variable list to pass</div><div style="font-family:Tahoma;font-size:13px"> -envnone do not pass any environment variables</div><div style="font-family:Tahoma;font-size:13px">
-envall pass all environment variables (default)</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Other local options:</div><div style="font-family:Tahoma;font-size:13px">
-n/-np {value} number of processes</div><div style="font-family:Tahoma;font-size:13px"> {exec_name} {args} executable name and arguments</div><div style="font-family:Tahoma;font-size:13px">
<br></div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Hydra specific options (treated as global):</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">
Launch options:</div><div style="font-family:Tahoma;font-size:13px"> -launcher launcher to use ( ssh rsh fork slurm ll lsf sge manual persist)</div><div style="font-family:Tahoma;font-size:13px">
-launcher-exec executable to use to launch processes</div><div style="font-family:Tahoma;font-size:13px"> -enable-x/-disable-x enable or disable X forwarding</div><div style="font-family:Tahoma;font-size:13px">
<br></div><div style="font-family:Tahoma;font-size:13px"> Resource management kernel options:</div><div style="font-family:Tahoma;font-size:13px"> -rmk resource management kernel to use ( user slurm ll lsf sge pbs)</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Hybrid programming options:</div><div style="font-family:Tahoma;font-size:13px"> -ranks-per-proc assign so many ranks to each process</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Processor topology options:</div><div style="font-family:Tahoma;font-size:13px"> -binding process-to-core binding mode</div>
<div style="font-family:Tahoma;font-size:13px"> -topolib processor topology library ( hwloc plpa)</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">
Checkpoint/Restart options:</div><div style="font-family:Tahoma;font-size:13px"> -ckpoint-interval checkpoint interval</div><div style="font-family:Tahoma;font-size:13px"> -ckpoint-prefix checkpoint file prefix</div>
<div style="font-family:Tahoma;font-size:13px"> -ckpoint-num checkpoint number to restart</div><div style="font-family:Tahoma;font-size:13px"> -ckpointlib checkpointing library (none)</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Demux engine options:</div><div style="font-family:Tahoma;font-size:13px"> -demux demux engine ( poll select)</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px"> Other Hydra options:</div><div style="font-family:Tahoma;font-size:13px"> -verbose verbose mode</div>
<div style="font-family:Tahoma;font-size:13px"> -info build information</div><div style="font-family:Tahoma;font-size:13px"> -print-all-exitcodes print exit codes of all processes</div>
<div style="font-family:Tahoma;font-size:13px"> -iface network interface to use</div><div style="font-family:Tahoma;font-size:13px"> -ppn processes per node</div>
<div style="font-family:Tahoma;font-size:13px"> -profile turn on internal profiling</div><div style="font-family:Tahoma;font-size:13px"> -prepend-rank prepend rank to output</div>
<div style="font-family:Tahoma;font-size:13px"> -prepend-pattern prepend pattern to output</div><div style="font-family:Tahoma;font-size:13px"> -outfile-pattern direct stdout to file</div>
<div style="font-family:Tahoma;font-size:13px"> -errfile-pattern direct stderr to file</div><div style="font-family:Tahoma;font-size:13px"> -nameserver name server information (host:port format)</div>
<div style="font-family:Tahoma;font-size:13px"> -disable-auto-cleanup don't cleanup processes on error</div><div style="font-family:Tahoma;font-size:13px"> -disable-hostname-propagation let MPICH2 auto-detect the hostname</div>
<div style="font-family:Tahoma;font-size:13px"> -order-nodes order nodes as ascending/descending cores</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">
Please see the intructions provided at</div><div style="font-family:Tahoma;font-size:13px"><a href="http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager">http://wiki.mcs.anl.gov/mpich2/index.php/Using_the_Hydra_Process_Manager</a></div>
<div style="font-family:Tahoma;font-size:13px">for further details</div><div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">Job /usr/share/lsf/7.0/linux2.6-glibc2.3-x86_64/bin/mpich2_wrapper -a mpich2 -n 12 -f /etc/hosts -launcher ssh ./ccsm.exe</div>
<div style="font-family:Tahoma;font-size:13px"><br></div><div style="font-family:Tahoma;font-size:13px">TID HOST_NAME COMMAND_LINE STATUS TERMINATION_TIME</div><div style="font-family:Tahoma;font-size:13px">
===== ========== ================ ======================= ===================</div><div style="font-family:Tahoma;font-size:13px">00001 cn60 Undefined </div><div style="font-family:Tahoma;font-size:13px">
"</div></div>