rank = 0 hostname = student1-ib0 Slave 1 finished Slave 3 finished GES_FTMPIService::WaitanyC - Fail of index 1 ========================================== Master::SendReceiveAsyncP2MP_C : ------------------------------------------ ErrorCode: 0x1C11B80F ErrorString: Other MPI error, error stack: PMPI_Test(170)............: MPI_Test(request=0x1b78ee64, flag=0x7fffd57c12cc, status=0x7fffd57c1330) failed MPIR_Test_impl(65)........: dequeue_and_set_error(597): Communication error with rank 2 Status.MPI_ERROR(code) = 460910128 Status.MPI_ERROR(str) = Invalid keyval, error stack: (unknown)(): ioctl failed Status.MPI_SOURCE = 0 Status.MPI_TAG = 0 Status.count = 0 Status.cancelled = 0 ========================================== Slave 4 finished Slave 5 finished Slave 6 finished Slave 7 finished Slave 8 finished Slave 9 finished Slave 1 iters 1000 - Sent exit message to slave rank 1 Slave 2 iters 20 - failed Slave 3 iters 1000 - Sent exit message to slave rank 3 Slave 4 iters 1000 - Sent exit message to slave rank 4 Slave 5 iters 1000 - Sent exit message to slave rank 5 Slave 6 iters 1000 - Sent exit message to slave rank 6 Slave 7 iters 1000 - Sent exit message to slave rank 7 Slave 8 iters 1000 - Sent exit message to slave rank 8 Slave 9 iters 1000 - Sent exit message to slave rank 9 End of master subroutine Sleep 2 secs ... Start abort