tput: No value for $TERM and no -T specified Rank 15 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 5 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 4 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x4c48280, rbuf=0x4c46150, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000002) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 5 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 8 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 14 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x4c17420, rbuf=0x4aed390, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000002) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 15 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 0 [Thu Jun 15 14:42:22 2017] [c3-0c0s15n3] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x4c50f90, rbuf=0x4b52490, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000004) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 8 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective _pmiu_daemon(SIGCHLD): [NID 00639] [c3-0c0s15n3] [Thu Jun 15 14:42:22 2017] PE RANK 8 exit signal Aborted [NID 00639] 2017-06-15 14:42:22 Apid 3590914: initiated application termination