[mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1

Wed Apr 20 12:16:02 CDT 2011

I also see the same problem when running just on two hosts and one core on each host.  The error message is a bit shorter (which I'll place at the end of this message).  When running the executable where mpich2 v1.2.1p1 is installed, there is no error (and the executable built against 1.2.1 or 1.3.2 ran fine).

Hope this helps.

cheers, roy

run (with my own debug info) and error message:
-----------------------------------------------
D:\roy>smpd -version
1.3.2p1

D:\roy>mpiexec -hosts 2 usctap3800 1 usctap3826 1 \\usdata011\MPRI-App\BAT\RoyTe
st\CorrelateMPI\CorrelateMPI.exe correlate_rep_traits.xml gwat_all_attieeric.h5
out.h5 debug
>>> Root process on computer: USCTAP3800
>>> Root process on computer: USCTAP3800
>>> No. of computers: 2
>>> Summary of InData
        Cfg file: correlate_rep_traits.xml
        Input hdf5 file: gwat_all_attieeric.h5
        Name of x data: repData
        Name of x ids: repIDs
        Name of y data: traitData
        Name of y ids: traitIDs
        Filter: pvalue
        Filter threshold: 1.0001
        Dataset name for correlation: correlations
        Dataset name for pvalue: pvalues
        Metric name: pearson
        Compression: 0
>>> Rank 0: reading input file: gwat_all_attieeric.h5
>>> File: out.h5 exists.
###
### Perf: Time to input data: 0 mins   2 secs
###
>>> in x data - Rank: 0 rows/cols/total: 39558/506/20016348
>>> in y data - Rank: 0 rows/cols/total: 347/506/175582
>>> Rank 0: Broadcasting input data to worker nodes
>>> rank: 1 metric: pearson Length: 7
>>> in x data - Rank: 1 rows/cols/total: 39558/506/20016348
>>> in y data - Rank: 1 rows/cols/total: 347/506/175582
Fatal error in PMPI_Bcast: Other MPI error, error stack:
PMPI_Bcast(1430).................................: MPI_Bcast(buf=00000000018C004
0, count=20016348, MPI_FLOAT, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast_impl(1273)............................:
MPIR_Bcast_intra(1107)...........................:
MPIR_Bcast_binomial(143).........................:
MPIC_Recv(110)...................................:
MPIC_Wait(540)...................................:
MPIDI_CH3I_Progress(353).........................:
MPID_nem_mpich2_blocking_recv(905)...............:
MPID_nem_newtcp_module_poll(37)..................:
MPID_nem_newtcp_module_connpoll(2669)............:
MPID_nem_newtcp_module_recv_success_handler(2364):
MPID_nem_newtcp_module_post_readv_ex(330)........:
MPIU_SOCKW_Readv_ex(392).........................: read from socket failed, An o
peration on a socket could not be performed because the system lacked sufficient
 buffer space or because a queue was full.
 (errno 10055)

job aborted:
rank: node: exit code[: error message]
0: usctap3800: 123
1: usctap3826: 1: process 1 exited without calling finalize

-----Original Message-----
From: Jayesh Krishna [mailto:jayesh at mcs.anl.gov] 
Sent: Tuesday, April 19, 2011 4:18 PM
To: mpich-discuss at mcs.anl.gov
Cc: Kuraisa, Roy J (BOSTON)
Subject: Re: [mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1

Hi,
 Can you send us the complete error message ?

Regards,
Jayesh

----- Original Message -----
From: "Roy J Kuraisa (BOSTON)" <roy_kuraisa at merck.com>
To: mpich-discuss at mcs.anl.gov
Sent: Tuesday, April 19, 2011 2:32:24 PM
Subject: [mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1

Hi, 
Summary: 
--------------- 
On Windows when I execute the following command (working on a fairly large dataset): 
mpiexec -hosts 2 usctap3825 15 usctap3488 1 \\fs1\correlatempi.exe cfg.xml in.h5 out.h5 debug 
I encounter an MPI gather error (read from socket failed (errno 10055). See error stack at end of this message. If I run on only one computer (with 16 cores): 
mpiexec -hosts 1 usctap3825 15 \\fs1\correlatempi.exe cfg.xml in.h5 out.h5 debug 
the program runs successfully. 

Additionally, both of the above commands run successfully on mpich2 v1.2.1 (although I had to build on mpich2 1.2.1 and used different servers that are configured exactly like the origian servers noted above (e.g., usctap3825, 16-core, 64GB memory, etc). 

I noticed that a similar error was fixed in mpich2-1.2 ( http://trac.mcs.anl.gov/projects/mpich2/ticket/895 ). Could this have regressed? tia. 

System Configuration: 
-------------------------------- 
Server1 (usctap3825) 
------- 
a. Windows Server 2003, 64-bit, SP2 
b. 16 cores/processors 
c. 64GB memory 
d. Physical computer 
Server2 (usctap3488) 
------- 
a. Windows Server 2003, 64-bit, SP2 
b. 2 cores/processors 
c. 8GB memory 
d. Virtual Machine 

cheers, roy 

error stack: 
---------------- 
Fatal error in PMPI_Gatherv: Other MPI error, error stack: 
PMPI_Gatherv(398)................................: MPI_Gatherv failed(sbuf=00000 
0003AA30040, scount=97787376, MPI_FLOAT, rbuf=0000000180040040, rcnts=000000000D 
6515E0, displs=000000000D651630, MPI_FLOAT, root=0, MPI_COMM_WORLD) failed 
MPIR_Gatherv_impl(210)...........................: 
MPIR_Gatherv(118)................................: 
MPIC_Waitall_ft(852).............................: 
MPIR_Waitall_impl(121)...........................: 
MPIDI_CH3I_Progress(353).........................: 
MPID_nem_mpich2_blocking_recv(905)...............: 
MPID_nem_newtcp_module_poll(37)..................: 
MPID_nem_newtcp_module_connpoll(2669)............: 
MPID_nem_newtcp_module_recv_success_handler(2364): 
MPID_nem_newtcp_module_post_readv_ex(330)........: 
MPIU_SOCKW_Readv_ex(392).........................: read from socket failed, An o 
peration on a socket could not be performed because the system lacked sufficient 
buffer space or because a queue was full. 
(errno 10055) 
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system. 
_______________________________________________
mpich-discuss mailing list
mpich-discuss at mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.