[mpich-discuss] Too many open files (errno 24) when using MPI_Alltoallv

[Dave Goodell]

Your OS has too low of a limit on the number of file descriptors that can be opened by a single process at a time (probably 1024).  Try increasing this limit.  Some instructions for Linux can be found here (just try step 1), you'll have to search around for instructions for other operating systems:

http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

-Dave

On Jun 7, 2012, at 1:16 AM CDT, jt.meng at siat.ac.cn wrote:

> I have trouble on using MPI_alltoallv over 1000 cores, I can not find out why this error come out， can you help me? 
> 
> The TestAlltoAll.cpp is attached in this email, You can have a try on 1000 cores to reproduce this errors, Thanks.
> 
> 
> 
> node73:/lustrefs/home/temp/BGI/newGraph/MPIGraph # mpic++ -O2 TestAlltoAll.cpp  -o TestAlltoAll
> node73:/lustrefs/home/temp/BGI/newGraph/MPIGraph # time mpirun -np 1056 -machinefile hostfile ./TestAlltoAll
> Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
> PMPI_Alltoallv(549)..............: MPI_Alltoallv(sbuf=0x2b2738837010, scnts=0x5f06e0, sdispls=0x5f89b0, MPI_LONG_LONG_INT, rbuf=0x2b276ade2010, rcnts=0x5f7920, rdispls=0x5f9a40, MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed
> MPIR_Alltoallv_impl(389).........:
> MPIR_Alltoallv(355)..............:
> MPIR_Alltoallv_intra(190)........:
> MPIC_Isend(475)..................:
> MPID_nem_lmt_RndvSend(81)........:
> MPIDI_CH3_RndvSend(63)...........: failure occurred while attempting to send RTS packet
> MPID_nem_tcp_iStartContigMsg(298):
> MPID_nem_tcp_connect(849)........: unable to create a socket, Too many open files (errno 24)
> Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
> PMPI_Alltoallv(549)..............: MPI_Alltoallv(sbuf=0x2b242dea0010, scnts=0x5f06e0, sdispls=0x5f89b0, MPI_LONG_LONG_INT, rbuf=0x2b246044b010, rcnts=0x5f7920, rdispls=0x5f9a40, MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed
> MPIR_Alltoallv_impl(389).........:
> MPIR_Alltoallv(355)..............:
> MPIR_Alltoallv_intra(190)........:
> MPIC_Isend(475)..................:
> MPID_nem_lmt_RndvSend(81)........:
> MPIDI_CH3_RndvSend(63)...........: failure occurred while attempting to send RTS packet
> MPID_nem_tcp_iStartContigMsg(298):
> MPID_nem_tcp_connect(849)........: unable to create a socket, Too many open files (errno 24)
> Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
> PMPI_Alltoallv(549)..............: MPI_Alltoallv(sbuf=0x2b1a26776010, scnts=0x5f06e0, sdispls=0x5f89b0, MPI_LONG_LONG_INT, rbuf=0x2b1a58d21010, rcnts=0x5f7920, rdispls=0x5f9a40, MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed
> MPIR_Alltoallv_impl(389).........:
> MPIR_Alltoallv(355)..............:
> MPIR_Alltoallv_intra(190)........:
> MPIC_Isend(475)..................:
> MPID_nem_lmt_RndvSend(81)........:
> MPIDI_CH3_RndvSend(63)...........: failure occurred while attempting to send RTS packet
> MPID_nem_tcp_iStartContigMsg(298):
> MPID_nem_tcp_connect(849)........: unable to create a socket, Too many open files (errno 24)
> Fatal error in PMPI_Alltoallv: Other MPI error, error stack:
> PMPI_Alltoallv(549)...............: MPI_Alltoallv(sbuf=0x2b7cd63af010, scnts=0x5f06e0, sdispls=0x5f89b0, MPI_LONG_LONG_INT, rbuf=0x2b7d0895a010, rcnts=0x5f7920, rdispls=0x5f9a40, MPI_LONG_LONG_INT, MPI_COMM_WORLD) failed
> MPIR_Alltoallv_impl(389)..........:
> MPIR_Alltoallv(355)...............:
> MPIR_Alltoallv_intra(199).........:
> MPIC_Waitall_ft(852)..............:
> MPIR_Waitall_impl(121)............:
> MPIDI_CH3I_Progress(402)..........:
> MPID_nem_mpich2_blocking_recv(905):
> MPID_nem_tcp_connpoll(1838).......:
> state_listening_handler(1908).....: accept of socket fd failed - Too many open files
> Ctrl-C caught... cleaning up processes
> 
> real    13m0.684s
> user    0m13.065s
> sys     0m49.703s
> 
> 
> Jintao Meng
> High Performance Computing Center
> Shenzhen Institutes of Advanced Technology, CAS
> 
> 
> 
> 
> 
> <TestAlltoAll.cpp>_______________________________________________
> mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
> To manage subscription options or unsubscribe:
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss