<P>Dear Pavan, </P>
<P> I have fixed this problem. Thanks very much for your help. </P>
<P> The file limitation on all computing nodes need to be updated by using the commond "echo "ulimit -n 32768" >> /etc/profile", and only the root will be affected by this modification.</P>
<P> Thanks,</P>
<P>Jintao <BR><BR><BR>> -----原始邮件-----<BR>> 发件人: "Pavan Balaji" <balaji@mcs.anl.gov><BR>> 发送时间: 2012年8月3日 星期五<BR>> 收件人: mpich-discuss@mcs.anl.gov<BR>> 抄送: jt.meng@siat.ac.cn<BR>> 主题: Re: [mpich-discuss] MPI error on MPI_alltoallv.<BR>> <BR>> <BR>> Please see this:<BR>> <BR>> https://lists.mcs.anl.gov/mailman/htdig/mpich-discuss/2012-June/012590.html<BR>> <BR>> -- Pavan<BR>> <BR>> On 08/02/2012 03:15 AM, jt.meng@siat.ac.cn wrote:<BR>> > Hi,<BR>> > My programs run well on 960 cores, howerver if it was running on<BR>> > 1024cores, I get the following error.<BR>> > I guess that this may cause by the OS limitations.<BR>> > Can anyone help me resolve this problem?<BR>> ><BR>> > ulimit output start here:<BR>> > ---------------------------------------------------------<BR>> > # ulimit -a<BR>> > core file size (blocks, -c) 0<BR>> > data seg size (kbytes, -d) unlimited<BR>> > file size (blocks, -f) unlimited<BR>> > pending signals (-i) 136192<BR>> > max locked memory (kbytes, -l) unlimited<BR>> > max memory size (kbytes, -m) unlimited<BR>> > open files &nbs p; (-n) 819200<BR>> > pipe size (512 bytes, -p) 8<BR>> > POSIX message queues (bytes, -q) 819200<BR>> > stack size (kbytes, -s) unlimited<BR>> > cpu time (seconds, -t) unlimited<BR>> > max user processes (-u) 136192<BR>> > virtual memory (kbytes, -v) unlimited<BR>> > file locks (-x) unlimited<BR>> ><BR>> ><BR>> > Error logs start here:<BR>> > ------------------------------------------------------------------------------------<BR>> > Fatal error in PMPI_Alltoallv: Other MPI error, error stack:<BR>> > PMPI_Alltoallv(549)...........: MPI_Alltoallv(sbuf=0x2b08c2bd7010 ,<BR>> > scnts=0x64ac20, sdispls=0x659b40, MPI_LONG_LONG_INT,<BR>> > rbuf=0x2b08c5bde010, rcnts=0x658b30, rdispls=0x65ab50,<BR>> > MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed<BR>> > MPIR_Alltoallv_impl(389)......:<BR>> > MPIR_Alltoallv(355)...........:<BR>> > MPIR_Alltoallv_intra(199).....:<BR>> > MPIC_Waitall_ft(852)..........:<BR>> > MPIR_Waitall_impl(121)........:<BR>> > MPIDI_CH3I_Progress(402)......:<BR>> > MPID_nem_mpich2_test_recv(747):<BR>> > MPID_nem_tcp_connpoll(1838)...:<BR>> > state_listening_handler(1908).: accept of socket fd failed - Too many<BR>> > open files<BR>> > Fatal error in PMPI_Alltoallv: Other MPI error, error stack:<BR>> > PMPI_Alltoallv(549)...........: MPI_Alltoallv(sbuf=0x2b974c333010,<BR>> > scnts=0x64ac20, sdispls=0x659b40, MPI_LONG_LONG_INT,<BR>> > rbuf=0x2b974f335010, rcnts=0x658b30, rdispls=0x65ab50,<BR>> > MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed<BR>> > MPIR_Alltoallv_impl(389)......:<BR>> > MPIR_Alltoallv(355)...........:<BR>> > MPIR_Alltoallv_int ra(199).....:<BR>> > MPIC_Waitall_ft(852)..........:<BR>> > MPIR_Waitall_impl(121)........:<BR>> > MPIDI_CH3I_Progress(402)......:<BR>> > MPID_nem_mpich2_test_recv(747):<BR>> > MPID_nem_tcp_connpoll(1838)...:<BR>> > state_listening_handler(1908).: accept of socket fd failed - Too many<BR>> > open files<BR>> > [proxy:0:9@node15] handle_pmi_response (./pm/pmiserv/pmip_cb.c:406):<BR>> > assert (!closed) failed<BR>> > [proxy:0:9@node15] HYD_pmcd_pmip_control_cmd_cb<BR>> > (./pm/pmiserv/pmip_cb.c:952): unable to handle PMI response<BR>> > [proxy:0:9@node15] HYDT_dmxu_poll_wait_for_event<BR>> > (./tools/demux/demux_poll.c:77): callback returned error status<BR>> > [proxy:0:9@node15] main (./pm/pmiserv/pmip.c:226): demux engine error<BR>> > waiting for event<BR>> > [mpiexec@node73] control_cb (./pm/pmiserv/pmiserv_cb.c:215): assert<BR>> > (!closed) failed<BR>> > [mpiexec@node73] HYDT_dmxu_poll_wait_for_event<BR>> > (./tools/demux/demux_poll.c:77): callback returned error status<BR>> > [mpiex ec@node73] HYD_pmci_wait_for_completion<BR>> > (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event<BR>> > [mpiexec@node73] main (./ui/mpich/mpiexec.c:405): process manager error<BR>> > waiting for completion<BR>> ><BR>> > Jintao<BR>> ><BR>> ><BR>> ><BR>> ><BR>> ><BR>> ><BR>> > _______________________________________________<BR>> > mpich-discuss mailing list mpich-discuss@mcs.anl.gov<BR>> > To manage subscription options or unsubscribe:<BR>> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss<BR>> ><BR>> <BR>> -- <BR>> Pavan Balaji<BR>> http://www.mcs.anl.gov/~balaji<BR><BR><BR><SPAN><BR>--<BR>- - - - - - - - - - - - - - - - - <BR><BR>孟金涛 工程师<BR>数字所 高性能中心<BR>中国科学院深圳先进技术研究院<BR><BR>地址：深圳市南山区西丽大学城学苑大道1068号<BR>电话：0755-86392368，13510470517<BR>邮编：518055<BR>2011-06-01</SPAN><BR></P><br><br><br>