<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have been trying to get mpiexec to run across nodes and it kept failing with:<o:p></o:p></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">/opt/mpich2/1.4.1p1/intel12/bin/hydra_pmi_proxy: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">HYD_pmcd_pmiserv_send_signal (./pm/pmiserv/pmiserv_cb.c:184): assert (!closed) failed</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">ui_cmd_cb (./pm/pmiserv/pmiserv_pmci.c:74): unable to send SIGUSR1 downstream</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Now I did troubleshoot this to be due to the fact that on the machines other than the one I am running mpiexec from do not have a proper LD_LIBRARY_PATH set.<o:p></o:p></p>
<p class="MsoNormal">Shouldn’t that get passed to the ‘children’ when I do an mpiexec?<o:p></o:p></p>
<p class="MsoNormal">We use modules and have different compilers here, so folks need to load the appropriate module to access the appropriate libraries. In this case, the application was compiled using the intel compiler, so the environment gets set using:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">module load compile/intel mpi/mpich2<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To get the application to run via mpich2 on multiple nodes, I have had to add that line to the .bashrc<o:p></o:p></p>
<p class="MsoNormal">I have tried using the various genv and env options (genvall, envall, envlist, genvlist) to no avail. The only way I was able to successfully run was to set the LD_LIBRARY_PATH in my .bashrc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Am I missing something about the environment settings when using mpiexec or mpirun?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Brian Andrus<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">ITACS/Research Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Naval Postgraduate School<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Monterey, California<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">voice: 831-656-6238<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>