<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=FR link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span lang=EN-US>Hello, <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I have an error using <b>mpiexec (MPICH2
1.4.1p)</b>. Hope somebody may help. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>The crash is random, i.e. the same
executable may crash or not.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><b><span lang=EN-US>Context</span></b><span lang=EN-US>:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>5 nodes heterogeneous cluster:<o:p></o:p></span></p>
<p class=MsoNormal style='text-indent:35.4pt'><span lang=EN-US>4 nodes with
CARMA (CUDA on ARM) on Ubuntu 11.4: the carrier board basically consists of an
ARM Cortex A9 processor and a Quadro 1000M NVIDIA GPU card.<o:p></o:p></span></p>
<p class=MsoNormal style='text-indent:35.4pt'><span lang=EN-US>1 node with one
XEON E5620 processor on Windows XP + cygwin.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Standard Ethernet network.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Names of the 5 nodes:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>lnardi <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>carma1<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>carma2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>carma3<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>carma4<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><b><span lang=EN-US>The command line</span></b><span
lang=EN-US> on the master node lnardi (Windows node) is:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>mpiexec -channel sock -n 1 -host lnardi
a.out : <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>-n 1 -host carma1 -path /home/lnardi/ a.out
: <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>-n 1 -host carma2 -path /home/lnardi/ a.out
: <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>-n 1 -host carma3 -path /home/lnardi/ a.out
: <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>-n 1 –host carma4 -path /home/lnardi/
a.out<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><b><span lang=EN-US>Notice that the same sample runs on a
full linux cluster </span></b><span lang=EN-US>with the following
characteristics: MVAPICH2-1.8a1p1 (mpirun) + MELLANOX infiniband + XEON X5675 +
NVIDIA GPUs M2090 + Red Hat Enterprise Linux Server release 6.2.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I was running a more complicated code but I
have reproduced the error on a trivial code:<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>#include</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#A31515'><mpi.h><o:p></o:p></span></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>#include</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#A31515'><stdio.h><o:p></o:p></span></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>#include</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#A31515'><string.h><o:p></o:p></span></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:#A31515'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>#define</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>BUFSIZE</span> 128<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>#define</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>TAG</span> 0<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New";color:blue'>int</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>main</span>(<span style='color:blue'>int</span> <span
style='color:#010001'>argc</span>, <span style='color:blue'>char</span> *<span
style='color:#010001'>argv</span>[])<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>{<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>char</span> <span style='color:#010001'>idstr</span>[32];<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>char</span> <span style='color:#010001'>buff</span>[<span
style='color:#010001'>BUFSIZE</span>];<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>int</span> <span style='color:#010001'>numprocs</span>;<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>int</span> <span style='color:#010001'>myid</span>;<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>int</span> <span style='color:#010001'>i</span>;<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>MPI_Status</span> <span style='color:#010001'>stat</span>;<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>MPI_Init</span>(&<span style='color:#010001'>argc</span>,&<span
style='color:#010001'>argv</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>MPI_Comm_size</span>(<span style='color:#010001'>MPI_COMM_WORLD</span>,&<span
style='color:#010001'>numprocs</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>MPI_Comm_rank</span>(<span style='color:#010001'>MPI_COMM_WORLD</span>,&<span
style='color:#010001'>myid</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>if</span>(<span style='color:#010001'>myid</span> == 0)<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> {<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>printf</span>(<span style='color:#A31515'>"%d:
We have %d processors\n"</span>, <span style='color:#010001'>myid</span>, <span
style='color:#010001'>numprocs</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:blue'>for</span>(<span style='color:#010001'>i</span>=1;<span
style='color:#010001'>i</span><<span style='color:#010001'>numprocs</span>;<span
style='color:#010001'>i</span>++)<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
{<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>sprintf</span>(<span style='color:#010001'>buff</span>,
<span style='color:#A31515'>"Hello %d! "</span>, <span
style='color:#010001'>i</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>MPI_Send</span>(<span style='color:#010001'>buff</span>,
<span style='color:#010001'>BUFSIZE</span>, <span style='color:#010001'>MPI_CHAR</span>,
<span style='color:#010001'>i</span>, <span style='color:#010001'>TAG</span>, <span
style='color:#010001'>MPI_COMM_WORLD</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
}<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:blue'>for</span>(<span style='color:#010001'>i</span>=1;<span
style='color:#010001'>i</span><<span style='color:#010001'>numprocs</span>;<span
style='color:#010001'>i</span>++)<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
{<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>MPI_Recv</span>(<span style='color:#010001'>buff</span>,
<span style='color:#010001'>BUFSIZE</span>, <span style='color:#010001'>MPI_CHAR</span>,
<span style='color:#010001'>i</span>, <span style='color:#010001'>TAG</span>, <span
style='color:#010001'>MPI_COMM_WORLD</span>, &<span style='color:#010001'>stat</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>printf</span>(<span style='color:#A31515'>"%d:
%s\n"</span>, <span style='color:#010001'>myid</span>, <span
style='color:#010001'>buff</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
}<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> }<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>else<o:p></o:p></span></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> {<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>MPI_Recv</span>(<span style='color:#010001'>buff</span>,
<span style='color:#010001'>BUFSIZE</span>, <span style='color:#010001'>MPI_CHAR</span>,
0, <span style='color:#010001'>TAG</span>, <span style='color:#010001'>MPI_COMM_WORLD</span>,
&<span style='color:#010001'>stat</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>sprintf</span>(<span style='color:#010001'>idstr</span>,
<span style='color:#A31515'>"Processor %d "</span>, <span
style='color:#010001'>myid</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>strncat</span>(<span style='color:#010001'>buff</span>,
<span style='color:#010001'>idstr</span>, <span style='color:#010001'>BUFSIZE</span>-1);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>strncat</span>(<span style='color:#010001'>buff</span>,
<span style='color:#A31515'>"reporting for duty\n"</span>, <span
style='color:#010001'>BUFSIZE</span>-1);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>
<span style='color:#010001'>MPI_Send</span>(<span style='color:#010001'>buff</span>,
<span style='color:#010001'>BUFSIZE</span>, <span style='color:#010001'>MPI_CHAR</span>,
0, <span style='color:#010001'>TAG</span>, <span style='color:#010001'>MPI_COMM_WORLD</span>);<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> }<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:#010001'>MPI_Finalize</span>();<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'> <span
style='color:blue'>return</span> 0;<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span lang=EN-US
style='font-size:10.0pt;font-family:"Courier New"'>}<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>The error:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: We have 5 processors<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: Hello 1! Processor 1 reporting for duty<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: Hello 2! Processor 2 reporting for duty<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: Hello 3! Processor 3 reporting for duty<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: Hello 4! Processor 4 reporting for duty<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>job aborted:<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>rank: node: exit code[: error message]<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>0: lnardi: -1073741819: process 0 exited
without calling finalize<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>1: carma1: -2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>2: carma2: -2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>3: carma3: -2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>4: carma4: -2<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I guess the problem comes from either the
sock channel or mpiexec or ARM. <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>What do you think about?<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Thanks <o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Dr Luigi Nardi<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
<p class="ecxMsoNormal" style="line-height: 19px; margin: 0px 0px 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; color: #2a2a2a; ">*******************************</p><p class="ecxMsoNormal" style="line-height: 19px; margin: 0px 0px 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; color: #2a2a2a; "><span lang="EN-GB" style="line-height: 17px; font-size: 10pt; color: black; font-family: Arial; ">This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorised to distribute, copy or use this e-mail or any attachment to it. </span><span style="font-family: Arial" color="black" size="2"><span style="line-height: 17px; font-size: 10pt; ">Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.</span></span></p></body>
</html>