<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">Hello,</span></font><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></div><div><div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">Le 27 déc. 2011 à 06:52, Pavan Balaji a écrit :</span></font></div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br class="Apple-interchange-newline"></span></font><blockquote type="cite"><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br>On 12/17/2011 02:56 AM, Bernard Chambon wrote:<br></span></font><blockquote type="cite"><blockquote type="cite"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">mpiexec -np 160 bin/basic_test<br></span></font></blockquote></blockquote><blockquote type="cite"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">Assertion failed in file<br></span></font></blockquote><blockquote type="cite"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">/scratch/BC/mpich2-1.4.1p1/src/util/wrappers/mpiu_shm_wrappers.h at line<br></span></font></blockquote><blockquote type="cite"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">889: seg_sz > 0<br></span></font></blockquote><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br>Looks like the shared memory is bombing out. Can you run mpiexec with the -verbose option and also send us the machine file that you are using (or is it all on a single node)?<br><br> -- Pavan<br><br>-- <br>Pavan Balaji<br><a href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a><br></span></font></div></blockquote></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">I run again my test (MPI_Init + MPI_Comm_rank + MPI_Comm_size + MPI_Finalize), on a single node</span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">and after :</span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> 1/ increasing </span></font><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">__FD_SETSIZE (1024 -> 8192) and recompiling mpich2 1.4 </span></font></span></div><div><span class="Apple-style-span" style="white-space: pre; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></span></div><div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">>grep -E "#define\W+__FD_SETSIZE" /usr/include/*.h /usr/include/*/*.h</span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">/usr/include/bits/typesizes.h:#define</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> </span></font></span><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">__FD_SETSIZE 8192</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> </span></font></span></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">/usr/include/linux/posix_types.h:#define __FD_SETSIZE</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> </span></font></span><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> 8192</span></font><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> </span></font></span></div></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"> 2/ asking my sysadmin to increase some limits</span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">>limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 10240 kbytes
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 4096
memorylocked 32 kbytes
maxproc 409600
</span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">>more /proc/sys/kernel/shmall
8388608
</span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;"><br></span></font></span></div><div><span class="Apple-style-span" style="white-space: pre;"><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">I got the same error when reaching a limit around 160 tasks (It's ok with, let say, 150 tasks)</span></font></span></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;">> mpiexec -verbose -np 160 bin/advance_test</span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;">….</span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;"><br></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;"><div>[proxy:0:0@ccwpge0001] got pmi command (from 114): get_my_kvsname</div><div><br></div><div>[proxy:0:0@ccwpge0001] PMI response: cmd=my_kvsname kvsname=kvs_10405_0</div><div>[proxy:0:0@ccwpge0001] got pmi command (from 8): barrier_in</div><div>[proxy:0:0@ccwpge0001] got pmi command (from 45): barrier_in</div><div>[proxy:0:0@ccwpge0001] got pmi command (from 84): barrier_in</div><div>[proxy:0:0@ccwpge0001] got pmi command (from 114): get</div><div>kvsname=kvs_10405_0 key=PMI_process_mapping </div><div>[proxy:0:0@ccwpge0001] PMI response: cmd=get_result rc=0 msg=success value=(vector,(0,1,1))</div><div>Assertion failed in file /scratch/BC/mpich2-1.4.1p1/src/util/wrappers/mpiu_shm_wrappers.h at line 889: seg_sz > 0</div><div>internal ABORT - process 0</div><div><br></div></span></font></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre; ">Best regards, </span></font><span class="Apple-style-span" style="font-size: 14px; ">and happy new year</span></div><div><span class="Apple-style-span" style="font-size: 14px; "><br></span></div><div><span class="Apple-style-span" style="font-size: 14px; ">PS : </span></div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; white-space: pre;"> To be clear, the purpose of this test is to understand why such a limit and,</span></font></div><div><span class="Apple-style-span" style="font-size: 14px; white-space: pre; "> more precisaly what is the relationship between that limit and the machine|user|software configuration</span></div><div><span class="Apple-style-span" style="font-size: 14px; white-space: pre; "><br></span></div><div>
<div><div><div><div><div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; ">---------------</span><span class="Apple-style-span" style="font-size: 14px; "><br></span><span class="Apple-style-span" style="font-size: 14px; ">Bernard CHAMBON</span><span class="Apple-style-span" style="font-size: 14px; "><br></span><span class="Apple-style-span" style="font-size: 14px; ">IN2P3 / CNRS</span><span class="Apple-style-span" style="font-size: 14px; "><br></span><span class="Apple-style-span" style="font-size: 14px; ">04 72 69 42 18</span><span class="Apple-style-span" style="font-size: 14px; "><br></span></font></div></div></div></div></div><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px;">
</span></font></div>
<br></div></body></html>