[mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1
Jayesh Krishna
jayesh at mcs.anl.gov
Thu Apr 21 11:14:08 CDT 2011
"Insufficient buffer space " errors should be handled (at least to a certain extent) by the library. I have created a ticket for the issue https://trac.mcs.anl.gov/projects/mpich2/ticket/1477 .
-Jayesh
----- Original Message -----
From: "Roy J Kuraisa (BOSTON)" <roy_kuraisa at merck.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>, mpich-discuss at mcs.anl.gov
Sent: Wednesday, April 20, 2011 12:16:02 PM
Subject: RE: [mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1
I also see the same problem when running just on two hosts and one core on each host. The error message is a bit shorter (which I'll place at the end of this message). When running the executable where mpich2 v1.2.1p1 is installed, there is no error (and the executable built against 1.2.1 or 1.3.2 ran fine).
Hope this helps.
cheers, roy
run (with my own debug info) and error message:
-----------------------------------------------
D:\roy>smpd -version
1.3.2p1
D:\roy>mpiexec -hosts 2 usctap3800 1 usctap3826 1 \\usdata011\MPRI-App\BAT\RoyTe
st\CorrelateMPI\CorrelateMPI.exe correlate_rep_traits.xml gwat_all_attieeric.h5
out.h5 debug
>>> Root process on computer: USCTAP3800
>>> Root process on computer: USCTAP3800
>>> No. of computers: 2
>>> Summary of InData
Cfg file: correlate_rep_traits.xml
Input hdf5 file: gwat_all_attieeric.h5
Name of x data: repData
Name of x ids: repIDs
Name of y data: traitData
Name of y ids: traitIDs
Filter: pvalue
Filter threshold: 1.0001
Dataset name for correlation: correlations
Dataset name for pvalue: pvalues
Metric name: pearson
Compression: 0
>>> Rank 0: reading input file: gwat_all_attieeric.h5
>>> File: out.h5 exists.
###
### Perf: Time to input data: 0 mins 2 secs
###
>>> in x data - Rank: 0 rows/cols/total: 39558/506/20016348
>>> in y data - Rank: 0 rows/cols/total: 347/506/175582
>>> Rank 0: Broadcasting input data to worker nodes
>>> rank: 1 metric: pearson Length: 7
>>> in x data - Rank: 1 rows/cols/total: 39558/506/20016348
>>> in y data - Rank: 1 rows/cols/total: 347/506/175582
Fatal error in PMPI_Bcast: Other MPI error, error stack:
PMPI_Bcast(1430).................................: MPI_Bcast(buf=00000000018C004
0, count=20016348, MPI_FLOAT, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast_impl(1273)............................:
MPIR_Bcast_intra(1107)...........................:
MPIR_Bcast_binomial(143).........................:
MPIC_Recv(110)...................................:
MPIC_Wait(540)...................................:
MPIDI_CH3I_Progress(353).........................:
MPID_nem_mpich2_blocking_recv(905)...............:
MPID_nem_newtcp_module_poll(37)..................:
MPID_nem_newtcp_module_connpoll(2669)............:
MPID_nem_newtcp_module_recv_success_handler(2364):
MPID_nem_newtcp_module_post_readv_ex(330)........:
MPIU_SOCKW_Readv_ex(392).........................: read from socket failed, An o
peration on a socket could not be performed because the system lacked sufficient
buffer space or because a queue was full.
(errno 10055)
job aborted:
rank: node: exit code[: error message]
0: usctap3800: 123
1: usctap3826: 1: process 1 exited without calling finalize
-----Original Message-----
From: Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
Sent: Tuesday, April 19, 2011 4:18 PM
To: mpich-discuss at mcs.anl.gov
Cc: Kuraisa, Roy J (BOSTON)
Subject: Re: [mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1
Hi,
Can you send us the complete error message ?
Regards,
Jayesh
----- Original Message -----
From: "Roy J Kuraisa (BOSTON)" <roy_kuraisa at merck.com>
To: mpich-discuss at mcs.anl.gov
Sent: Tuesday, April 19, 2011 2:32:24 PM
Subject: [mpich-discuss] read from socket failed (errno 10055) on 1.3.2p1
Hi,
Summary:
---------------
On Windows when I execute the following command (working on a fairly large dataset):
mpiexec -hosts 2 usctap3825 15 usctap3488 1 \\fs1\correlatempi.exe cfg.xml in.h5 out.h5 debug
I encounter an MPI gather error (read from socket failed (errno 10055). See error stack at end of this message. If I run on only one computer (with 16 cores):
mpiexec -hosts 1 usctap3825 15 \\fs1\correlatempi.exe cfg.xml in.h5 out.h5 debug
the program runs successfully.
Additionally, both of the above commands run successfully on mpich2 v1.2.1 (although I had to build on mpich2 1.2.1 and used different servers that are configured exactly like the origian servers noted above (e.g., usctap3825, 16-core, 64GB memory, etc).
I noticed that a similar error was fixed in mpich2-1.2 ( http://trac.mcs.anl.gov/projects/mpich2/ticket/895 ). Could this have regressed? tia.
System Configuration:
--------------------------------
Server1 (usctap3825)
-------
a. Windows Server 2003, 64-bit, SP2
b. 16 cores/processors
c. 64GB memory
d. Physical computer
Server2 (usctap3488)
-------
a. Windows Server 2003, 64-bit, SP2
b. 2 cores/processors
c. 8GB memory
d. Virtual Machine
cheers, roy
error stack:
----------------
Fatal error in PMPI_Gatherv: Other MPI error, error stack:
PMPI_Gatherv(398)................................: MPI_Gatherv failed(sbuf=00000
0003AA30040, scount=97787376, MPI_FLOAT, rbuf=0000000180040040, rcnts=000000000D
6515E0, displs=000000000D651630, MPI_FLOAT, root=0, MPI_COMM_WORLD) failed
MPIR_Gatherv_impl(210)...........................:
MPIR_Gatherv(118)................................:
MPIC_Waitall_ft(852).............................:
MPIR_Waitall_impl(121)...........................:
MPIDI_CH3I_Progress(353).........................:
MPID_nem_mpich2_blocking_recv(905)...............:
MPID_nem_newtcp_module_poll(37)..................:
MPID_nem_newtcp_module_connpoll(2669)............:
MPID_nem_newtcp_module_recv_success_handler(2364):
MPID_nem_newtcp_module_post_readv_ex(330)........:
MPIU_SOCKW_Readv_ex(392).........................: read from socket failed, An o
peration on a socket could not be performed because the system lacked sufficient
buffer space or because a queue was full.
(errno 10055)
Notice: This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from
your system.
_______________________________________________
mpich-discuss mailing list
mpich-discuss at mcs.anl.gov
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
Notice: This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from
your system.
More information about the mpich-discuss
mailing list