2010-09-07-12-16 attrt 2 ./attr fail Fatal error in PMPI_Comm_dup: Other MPI error, error stack: PMPI_Comm_dup(176)................: MPI_Comm_dup(MPI_COMM_WORLD, new_comm=0x7fff1494d83c) failed PMPI_Comm_dup(161)................: MPIR_Comm_dup_impl(55)............: MPIR_Comm_copy(959)...............: MPIR_Get_contextid(517)...........: MPIR_Get_contextid_sparse(677)....: MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Comm_dup: Other MPI error, error stack: PMPI_Comm_dup(176)................: MPI_Comm_dup(MPI_COMM_WORLD, new_comm=0x7fff18785c1c) failed PMPI_Comm_dup(161)................: MPIR_Comm_dup_impl(55)............: MPIR_Comm_copy(959)...............: MPIR_Get_contextid(517)...........: MPIR_Get_contextid_sparse(677)....: MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: attric 4 ./attr fail Fatal error in PMPI_Comm_split: Other MPI error, error stack: PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=1, key=0, new_comm=0x7fff67254f9c) failed PMPI_Comm_split(400)..............: MPIR_Comm_split_impl(88)..........: MPIR_Allgather_impl(744)..........: MPIR_Allgather(705)...............: MPIR_Allgather_intra(177).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Comm_split: Other MPI error, error stack: PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=2, new_comm=0x7fffba96e51c) failed PMPI_Comm_split(400)..............: MPIR_Comm_split_impl(88)..........: MPIR_Allgather_impl(744)..........: MPIR_Allgather(705)...............: MPIR_Allgather_intra(177).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 3: Fatal error in PMPI_Comm_split: Other MPI error, error stack: PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=1, key=1, new_comm=0x7fff84aaa5fc) failed PMPI_Comm_split(400)..............: MPIR_Comm_split_impl(88)..........: MPIR_Allgather_impl(744)..........: MPIR_Allgather(705)...............: MPIR_Allgather_intra(177).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: Fatal error in PMPI_Comm_split: Other MPI error, error stack: PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=3, new_comm=0x7fffe475353c) failed PMPI_Comm_split(400)..............: MPIR_Comm_split_impl(88)..........: MPIR_Allgather_impl(744)..........: MPIR_Allgather(705)...............: MPIR_Allgather_intra(177).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 2: attrerr 1 ./attr pass attrend 1 ./attr pass attrend 4 ./attr fail Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(302).................: MPI_Finalize failed MPI_Finalize(210).................: MPID_Finalize(88).................: MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(302).................: MPI_Finalize failed MPI_Finalize(210).................: MPID_Finalize(88).................: MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) attrend2 1 ./attr pass attrend2 5 ./attr fail Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(302).................: MPI_Finalize failed MPI_Finalize(210).................: MPID_Finalize(88).................: MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(302).................: MPI_Finalize failed MPI_Finalize(210).................: MPID_Finalize(88).................: MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) attrerrcomm 1 ./attr pass attrerrtype 1 ./attr pass attr2type 1 ./attr pass attrorder 1 ./attr pass attrordercomm 1 ./attr pass attrordertype 1 ./attr pass baseattr2 1 ./attr pass baseattrcomm 1 ./attr pass fkeyval 1 ./attr pass fkeyvalcomm 1 ./attr pass fkeyvaltype 1 ./attr pass keyval_double_free 1 ./attr pass allred 4 ./coll fail Error class 15 (Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x1577dfb0, rbuf=0x1577dfe0, count=10, MPI_INT, MPI_SUM, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: ) (0) Error for type MPI_INT and op MPI_SUM Error class 15 (Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x1f88efb0, rbuf=0x1f88efe0, count=10, MPI_INT, MPI_SUM, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: ) (1) Error for type MPI_INT and op MPI_SUM APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1) allredmany 4 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x7fffdbdcef80, rbuf=0x7fffdbdcef78, count=1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x7fffbf3a7b40, rbuf=0x7fffbf3a7b38, count=1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allred2 4 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff1f30688, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3d33d918, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allred3 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff64258548, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffc236b538, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allred4 4 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff4ce02788, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff5773e68, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) reduce 5 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffbad28fa8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffdaa71db8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) reduce 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff77a903b8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3501b118, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) reduce_local 2 ./coll fail Fatal error in PMPI_Reduce: Other MPI error, error stack: PMPI_Reduce(1207).................: MPI_Reduce(sbuf=0x7fffdc6181ac, rbuf=0x7fffdc6181b4, count=1, MPI_INT, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1025)............: MPIR_Reduce_intra(855)............: MPIR_Reduce_binomial(206).........: MPIC_Send(66).....................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: [mpiexec@aramis] HYDT_dmx_deregister_fd (./tools/demux/demux.c:142): could not find fd to deregister: -2 [mpiexec@aramis] HYD_pmcd_pmiserv_cleanup (./pm/pmiserv/pmiserv_cb.c:398): error deregistering fd APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1) op_commutative 2 ./coll fail Fatal error in PMPI_Reduce: Other MPI error, error stack: PMPI_Reduce(1207).................: MPI_Reduce(sbuf=0x7fff0861b0dc, rbuf=0x7fff0861b0e4, count=1, MPI_INT, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1025)............: MPIR_Reduce_intra(855)............: MPIR_Reduce_binomial(206).........: MPIC_Send(66).....................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: [mpiexec@aramis] HYDT_dmx_deregister_fd (./tools/demux/demux.c:142): could not find fd to deregister: -2 [mpiexec@aramis] HYD_pmcd_pmiserv_cleanup (./pm/pmiserv/pmiserv_cb.c:398): error deregistering fd APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1) red3 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff74c9678, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3db7a8d8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) red4 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff85728958, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff5dd47ff8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoall1 8 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff9ea63b88, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff1c1592c8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoallv 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff833e6bf8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffe7222138, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoallv0 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff6ca3b538, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff29ddc248, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoallw1 10 ./coll fail Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=0x11fbfae0, scnts=0x11fbf920, sdispls=0x11fbfce0, stypes=0x11fbfd40, rbuf=0x11fbfbe0, rcnts=0x11fbf950, rdispls=0x11fbfd10, rtypes=0x11fbfd70, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(168).........: MPIR_Waitall_impl(121)............: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 3: Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=0xb41fae0, scnts=0xb41f920, sdispls=0xb41fce0, stypes=0xb41fd40, rbuf=0xb41fbe0, rcnts=0xb41f950, rdispls=0xb41fd10, rtypes=0xb41fd70, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(168).........: MPIR_Waitall_impl(121)............: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 5: [mpiexec@aramis] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:99): one of the processes terminated badly; timing out [mpiexec@aramis] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:18): bootstrap device returned error waiting for completion [mpiexec@aramis] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:325): bootstrap server returned error waiting for completion [mpiexec@aramis] main (./ui/mpich/mpiexec.c:293): process manager error waiting for completion alltoallw2 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff844d0e8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffac5325f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoallw_zeros 1 ./coll pass alltoallw_zeros 2 ./coll fail Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff2408b218, rcnts=0x18582560, rdispls=0x185825a0, rtypes=0x18582500, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(108).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff9bba1b98, rcnts=0x1a644580, rdispls=0x1a6445c0, rtypes=0x1a644480, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(117).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: alltoallw_zeros 5 ./coll fail Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff168eff58, rcnts=0x183af400, rdispls=0x183b12d0, rtypes=0x183af340, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(108).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fffba7a2dc8, rcnts=0x808d420, rdispls=0x808d460, rtypes=0x808d260, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(117).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) alltoallw_zeros 8 ./coll fail Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff350d9538, rcnts=0x1ed121b0, rdispls=0x1ed12210, rtypes=0x1ed100a0, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(108).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Alltoallw: Other MPI error, error stack: PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff23653c48, rcnts=0xf4da1e0, rdispls=0xf4da240, rtypes=0xf4d80d0, MPI_COMM_WORLD) failed MPIR_Alltoallw_impl(353)..........: MPIR_Alltoallw(319)...............: MPIR_Alltoallw_intra(117).........: MPIC_Sendrecv_replace(271)........: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: [proxy:0:1@porthos] handle_pmi_response (./pm/pmiserv/pmip_cb.c:408): assert (!closed) failed [proxy:0:1@porthos] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:895): unable to handle PMI response [proxy:0:1@porthos] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@porthos] main (./pm/pmiserv/pmip.c:264): demux engine error waiting for event [mpiexec@aramis] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:70): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP)) failed [mpiexec@aramis] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:84): error waiting for event [mpiexec@aramis] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:18): bootstrap device returned error waiting for completion [mpiexec@aramis] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:325): bootstrap server returned error waiting for completion [mpiexec@aramis] main (./ui/mpich/mpiexec.c:293): process manager error waiting for completion allgather2 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff1dfdbc68, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff9435c838, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allgather3 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff550033c8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffd1b89df8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allgatherv2 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3a5680f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff030593d8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allgatherv3 10 ./coll fail Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff5dc3fd88, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in MPI_Allreduce: Other MPI error, error stack: MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff99fa96f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed MPIR_Allreduce_impl(684)..........: MPIR_Allreduce_intra(192).........: allreduce_intra_or_coll_fn(103)...: MPIR_Allreduce_intra(362).........: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) allgatherv4 4 ./coll fail Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(362).................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(895): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(362).................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(241)............: MPIR_Barrier_or_coll_fn(112)......: MPIR_Barrier_intra(79)............: MPIC_Sendrecv(186)................: MPIC_Wait(534)....................: MPIDI_CH3I_Progress(184)..........: MPID_nem_mpich2_blocking_recv(880): MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15) bcasttest 4 ./coll