[mpich-discuss] MPI error

Jayesh Krishna jayesh at mcs.anl.gov
Wed Apr 18 16:02:21 CDT 2012


Hi,
 One possibility is that rank 0 crashed (hence might not have called MPI_Finalize() & aborting the whole job).

Regards,
Jayesh

----- Original Message -----
From: "Qiguo Jing" <QJing at TrinityConsultants.com>
To: mpich-discuss at mcs.anl.gov
Sent: Wednesday, April 18, 2012 2:34:45 PM
Subject: [mpich-discuss]  MPI error


Hi there, 

Can someone help me on the following issue? The MPI program we are using works fine with single PC of all OS (from XP to Win7, 32 and 64). However, when we run it on a cluster, for some cases, we got the following error. We are using MPICH2-1.2.1-win-ia32. 

Thanks you. 

Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 

job aborted: 
rank: node: exit code[: error message] 
0: 10.0.0.52: 157: process 0 exited without calling finalize 
1: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
2: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
3: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
4: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
5: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
6: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
7: 10.0.0.52: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
8: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
9: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
10: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
11: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
12: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
13: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
14: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
15: 10.0.0.61: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
16: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
17: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
18: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
19: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
20: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
21: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
22: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
23: 10.0.0.59: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
24: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
25: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
26: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
27: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
28: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
29: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
30: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
31: 10.0.0.57: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
32: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
33: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
34: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
35: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
36: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
37: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
38: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
39: 10.0.0.60: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
40: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
41: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
42: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
43: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
44: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
45: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
46: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
47: 10.0.0.64: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
48: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
49: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
50: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
51: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
52: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
53: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
54: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
55: 10.0.0.63: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
56: 10.0.0.62: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
57: 10.0.0.62: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
58: 10.0.0.62: 1: Fatal error in MPI_Recv: Other MPI error, error stack: 
MPI_Recv(187).............................: MPI_Recv(buf=0012F93C, count=1, MPI_INTEGER, src=0, tag=4638, MPI_COMM_WORLD, status=00000001) failed 
MPIDI_CH3i_Progress_wait(215).............: an error occurred while handling an event returned by MPIDU_Sock_Wait() 
MPIDI_CH3I_Progress_handle_sock_event(420): 
MPIDU_Sock_wait(2606).....................: The specified network name is no longer available. (errno 64) 
59: 10.0.0.62: 1: Fatal error in MPI_Recv: Other MPI error, error stack: _________________________________________________________________________

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material.  Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited.   If you
Received this in error, please contact the sender and delete the material 
from any computer.
_________________________________________________________________________ 
_______________________________________________
mpich-discuss mailing list     mpich-discuss at mcs.anl.gov
To manage subscription options or unsubscribe:
https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss


More information about the mpich-discuss mailing list