tput: No value for $TERM and no -T specified Rank 9 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 8 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x49bf660, rbuf=0x48bb1d0, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000002) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 9 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 10 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 14 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x49bcd20, rbuf=0x49ba610, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000002) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 15 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 12 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x49c4b20, rbuf=0x49c2410, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000002) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 13 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 13 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 11 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 10 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 15 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 4 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 Rank 0 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPI_Allreduce: Message truncated, error stack: MPI_Allreduce(1000)...............: MPI_Allreduce(sbuf=0x49909d0, rbuf=0x48d7690, count=102, MPI_DOUBLE, MPI_SUM, comm=0x84000004) failed MPIR_Allreduce_impl(843)..........: MPIR_Allreduce_intra(272).........: MPIR_Reduce_impl(1200)............: MPIR_Reduce_intra(996)............: MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 2 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Reduce_binomial(233).........: MPIDI_CH3U_Receive_data_found(144): Message from rank 4 and tag 11 truncated; 824 bytes received but buffer size is 816 MPIR_Allreduce_intra(302).........: MPIR_Bcast_impl(1454).............: MPIR_CRAY_Bcast(1220).............: MPIR_CRAY_Bcast_Tree(470).........: Failure during collective Rank 2 [Thu Jun 15 09:20:41 2017] [c1-0c0s10n0] Fatal error in MPIR_CRAY_Bcast_Tree: Other MPI error, error stack: MPIR_CRAY_Bcast_Tree(405): message sizes do not match across processes in the collective routine: Received 102 but expected 103 _pmiu_daemon(SIGCHLD): [NID 00232] [c1-0c0s10n0] [Thu Jun 15 09:20:41 2017] PE RANK 12 exit signal Aborted [NID 00232] 2017-06-15 09:20:41 Apid 3590478: initiated application termination