<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Hello,</span></font><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Thank you Gustavo and thank you Dave for you interest in my problem,</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font><div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Le 16 déc. 2011 à 16:35, Gustavo Correa a écrit :</span></font></div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br class="Apple-interchange-newline"></span></font><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Hi Bernard<br><br>Am I mistaken, or does your main routine perhaps calls only</span></font><span class="Apple-converted-space"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br>MPI_Init?<br>Your main seems to call only 'basicTest', but not 'rank',<br>where other MPI routines appear.<br><br>The MPICH2 developers may shed some light here,<br>but I think MPI_Init alone doesn't compose a minimal MPI program.<br>You need at least MPI_Finalize, I guess.<br>Or not?<br><br></span></font></span></blockquote><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">NO, you probably miss an </span></font><span class="hps"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">opening brace</span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> "</span></font><span class="hps atn"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">{", (due to my bad indentation, sorry)</span></font></span></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">In fact my basicTest function include : MPI_Init + MPI_Comm_rank + MPI_Comm_size + MPI_Finalize</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><blockquote type="cite"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Also, not related to your C program, but</span></font><span class="Apple-converted-space"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br>since you are in Linux, why did you choose g77 to compile the Fortran-77 bindings,<br>and f95 [is this g95?] to compile the Fortran-90 bindings of MPICH2?<br>g77 is quite old, I have been luckier using gfortran to compile both</span></font><span class="Apple-converted-space"><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br>the Fortran 77 and 90 bindings.</span></font></blockquote><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">mpich2 1.0.x was the installed software on our machines, not by myself</span></font></div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">so I can tell you, at the current time, why g77 </span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">(A try with mpich2 1.0.x </span></font><span class="Apple-style-span" style="font-size: 15px; "> was just a reference to shown that I</span><span class="Apple-style-span" style="font-size: 15px; "> can have more than 512 tasks) </span></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">What I can tell is that, after compiling the latest mpich2 version (1.4.1p1)</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">I encountered the failure when number of tasks reached ~160 tasks</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">I take care why the number of file descriptors and also the sire of share memory (SHMALL)</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Here are the values :</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><div style="font-size: 11px; "><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">>mpich2version</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 Version: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">1.4.1p1</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 Release date:</span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Thu Sep 1 13:53:02 CDT 2011</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 Device: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">ch3:nemesis</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 configure: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">--prefix=//scratch/BC/mpich2-1.4</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 CC: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">/usr/bin/gcc -m64 -O2</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 CXX: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">c++ -m64 -O2</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 F77: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">/usr/bin/f77 -O2</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">MPICH2 FC: </span></font><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> </span></font></span><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">f95 </span></font></div></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> >limit</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">cputime unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">filesize unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">datasize unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">stacksize 10240 kbytes</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">coredumpsize unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">memoryuse unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">vmemoryuse unlimited</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">descriptors 4096 </span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">memorylocked 32 kbytes</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">maxproc 409600 </span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> >more /proc/sys/kernel/shmall</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">8388608</span></font></div></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Here is the test</span></font></div><div><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> >mpiexec -np 150 bin/basic_test</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Running 150 tasks </span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "> >mpiexec -np 160 bin/basic_test</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Assertion failed in file /scratch/BC/mpich2-1.4.1p1/src/util/wrappers/mpiu_shm_wrappers.h at line 889: seg_sz > 0</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">internal ABORT - process 0</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[proxy:0:0@ccwpge0001] send_cmd_downstream (./pm/pmiserv/pmip_pmi_v1.c:80): assert (!closed) failed</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[proxy:0:0@ccwpge0001] fn_get (./pm/pmiserv/pmip_pmi_v1.c:349): error sending PMI response</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[proxy:0:0@ccwpge0001] pmi_cb (./pm/pmiserv/pmip_cb.c:327): PMI handler returned error</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[proxy:0:0@ccwpge0001] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[proxy:0:0@ccwpge0001] main (./pm/pmiserv/pmip.c:226): demux engine error waiting for event</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[mpiexec@ccwpge0001] control_cb (./pm/pmiserv/pmiserv_cb.c:215): assert (!closed) failed</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[mpiexec@ccwpge0001] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[mpiexec@ccwpge0001] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">[mpiexec@ccwpge0001] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">As Dave pointed in an more recent mail, perhaps I must increase </span></font><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">__FD_SETSIZE , and reocmpile mpich2</span></font></span></div><div style="font-size: 11px; "><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">but I have to ask my sysadmin, for that !</span></font></span></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">The OS and CPU are : </span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><div>>uname -a </div><div>Linux ccwpge0001 2.6.18-238.12cc.el5 #1 SMP Thu Mar 3 12:19:21 CET 2011 x86_64 x86_64 x86_64 GNU/Linux</div><div><br></div><div><div>grep ... /proc/cpuinfo</div><div>model name<span class="Apple-tab-span" style="white-space:pre"> </span>: Intel(R) Xeon(R) CPU E5540 @ 2.53GHz</div></div><div><br></div></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 15px;"><br></span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">Best regards</span></font></div><div style="font-size: 11px; "><font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; "><br></span></font></div><div style="font-size: 11px; ">
<font class="Apple-style-span" size="5" style="font-size: 18px; "><span class="Apple-style-span" style="font-size: 15px; ">---------------<br>Bernard CHAMBON<br>IN2P3 / CNRS<br>04 72 69 42 18<br></span>
</font></div>
<br style="font-size: 11px; "></div></body></html>