2010-09-07-12-16
attrt
2
./attr
fail
Fatal error in PMPI_Comm_dup: Other MPI error, error stack:
PMPI_Comm_dup(176)................: MPI_Comm_dup(MPI_COMM_WORLD, new_comm=0x7fff1494d83c) failed
PMPI_Comm_dup(161)................:
MPIR_Comm_dup_impl(55)............:
MPIR_Comm_copy(959)...............:
MPIR_Get_contextid(517)...........:
MPIR_Get_contextid_sparse(677)....:
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Comm_dup: Other MPI error, error stack:
PMPI_Comm_dup(176)................: MPI_Comm_dup(MPI_COMM_WORLD, new_comm=0x7fff18785c1c) failed
PMPI_Comm_dup(161)................:
MPIR_Comm_dup_impl(55)............:
MPIR_Comm_copy(959)...............:
MPIR_Get_contextid(517)...........:
MPIR_Get_contextid_sparse(677)....:
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
attric
4
./attr
fail
Fatal error in PMPI_Comm_split: Other MPI error, error stack:
PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=1, key=0, new_comm=0x7fff67254f9c) failed
PMPI_Comm_split(400)..............:
MPIR_Comm_split_impl(88)..........:
MPIR_Allgather_impl(744)..........:
MPIR_Allgather(705)...............:
MPIR_Allgather_intra(177).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Comm_split: Other MPI error, error stack:
PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=2, new_comm=0x7fffba96e51c) failed
PMPI_Comm_split(400)..............:
MPIR_Comm_split_impl(88)..........:
MPIR_Allgather_impl(744)..........:
MPIR_Allgather(705)...............:
MPIR_Allgather_intra(177).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 3:
Fatal error in PMPI_Comm_split: Other MPI error, error stack:
PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=1, key=1, new_comm=0x7fff84aaa5fc) failed
PMPI_Comm_split(400)..............:
MPIR_Comm_split_impl(88)..........:
MPIR_Allgather_impl(744)..........:
MPIR_Allgather(705)...............:
MPIR_Allgather_intra(177).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
Fatal error in PMPI_Comm_split: Other MPI error, error stack:
PMPI_Comm_split(418)..............: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=3, new_comm=0x7fffe475353c) failed
PMPI_Comm_split(400)..............:
MPIR_Comm_split_impl(88)..........:
MPIR_Allgather_impl(744)..........:
MPIR_Allgather(705)...............:
MPIR_Allgather_intra(177).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 2:
attrerr
1
./attr
pass
attrend
1
./attr
pass
attrend
4
./attr
fail
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(302).................: MPI_Finalize failed
MPI_Finalize(210).................:
MPID_Finalize(88).................:
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(302).................: MPI_Finalize failed
MPI_Finalize(210).................:
MPID_Finalize(88).................:
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
attrend2
1
./attr
pass
attrend2
5
./attr
fail
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(302).................: MPI_Finalize failed
MPI_Finalize(210).................:
MPID_Finalize(88).................:
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(302).................: MPI_Finalize failed
MPI_Finalize(210).................:
MPID_Finalize(88).................:
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
attrerrcomm
1
./attr
pass
attrerrtype
1
./attr
pass
attr2type
1
./attr
pass
attrorder
1
./attr
pass
attrordercomm
1
./attr
pass
attrordertype
1
./attr
pass
baseattr2
1
./attr
pass
baseattrcomm
1
./attr
pass
fkeyval
1
./attr
pass
fkeyvalcomm
1
./attr
pass
fkeyvaltype
1
./attr
pass
keyval_double_free
1
./attr
pass
allred
4
./coll
fail
Error class 15 (Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x1577dfb0, rbuf=0x1577dfe0, count=10, MPI_INT, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1: )
(0) Error for type MPI_INT and op MPI_SUM
Error class 15 (Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x1f88efb0, rbuf=0x1f88efe0, count=10, MPI_INT, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0: )
(1) Error for type MPI_INT and op MPI_SUM
APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)
allredmany
4
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x7fffdbdcef80, rbuf=0x7fffdbdcef78, count=1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=0x7fffbf3a7b40, rbuf=0x7fffbf3a7b38, count=1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allred2
4
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff1f30688, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3d33d918, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allred3
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff64258548, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffc236b538, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allred4
4
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff4ce02788, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff5773e68, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
reduce
5
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffbad28fa8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffdaa71db8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
reduce
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff77a903b8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3501b118, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
reduce_local
2
./coll
fail
Fatal error in PMPI_Reduce: Other MPI error, error stack:
PMPI_Reduce(1207).................: MPI_Reduce(sbuf=0x7fffdc6181ac, rbuf=0x7fffdc6181b4, count=1, MPI_INT, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1025)............:
MPIR_Reduce_intra(855)............:
MPIR_Reduce_binomial(206).........:
MPIC_Send(66).....................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
[mpiexec@aramis] HYDT_dmx_deregister_fd (./tools/demux/demux.c:142): could not find fd to deregister: -2
[mpiexec@aramis] HYD_pmcd_pmiserv_cleanup (./pm/pmiserv/pmiserv_cb.c:398): error deregistering fd
APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)
op_commutative
2
./coll
fail
Fatal error in PMPI_Reduce: Other MPI error, error stack:
PMPI_Reduce(1207).................: MPI_Reduce(sbuf=0x7fff0861b0dc, rbuf=0x7fff0861b0e4, count=1, MPI_INT, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1025)............:
MPIR_Reduce_intra(855)............:
MPIR_Reduce_binomial(206).........:
MPIC_Send(66).....................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
[mpiexec@aramis] HYDT_dmx_deregister_fd (./tools/demux/demux.c:142): could not find fd to deregister: -2
[mpiexec@aramis] HYD_pmcd_pmiserv_cleanup (./pm/pmiserv/pmiserv_cb.c:398): error deregistering fd
APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)
red3
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff74c9678, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3db7a8d8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
red4
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff85728958, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff5dd47ff8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoall1
8
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff9ea63b88, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff1c1592c8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoallv
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff833e6bf8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffe7222138, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoallv0
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff6ca3b538, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff29ddc248, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoallw1
10
./coll
fail
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=0x11fbfae0, scnts=0x11fbf920, sdispls=0x11fbfce0, stypes=0x11fbfd40, rbuf=0x11fbfbe0, rcnts=0x11fbf950, rdispls=0x11fbfd10, rtypes=0x11fbfd70, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(168).........:
MPIR_Waitall_impl(121)............:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 3:
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=0xb41fae0, scnts=0xb41f920, sdispls=0xb41fce0, stypes=0xb41fd40, rbuf=0xb41fbe0, rcnts=0xb41f950, rdispls=0xb41fd10, rtypes=0xb41fd70, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(168).........:
MPIR_Waitall_impl(121)............:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 5:
[mpiexec@aramis] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:99): one of the processes terminated badly; timing out
[mpiexec@aramis] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:18): bootstrap device returned error waiting for completion
[mpiexec@aramis] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:325): bootstrap server returned error waiting for completion
[mpiexec@aramis] main (./ui/mpich/mpiexec.c:293): process manager error waiting for completion
alltoallw2
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7ffff844d0e8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffac5325f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoallw_zeros
1
./coll
pass
alltoallw_zeros
2
./coll
fail
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff2408b218, rcnts=0x18582560, rdispls=0x185825a0, rtypes=0x18582500, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(108).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff9bba1b98, rcnts=0x1a644580, rdispls=0x1a6445c0, rtypes=0x1a644480, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(117).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
alltoallw_zeros
5
./coll
fail
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff168eff58, rcnts=0x183af400, rdispls=0x183b12d0, rtypes=0x183af340, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(108).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fffba7a2dc8, rcnts=0x808d420, rdispls=0x808d460, rtypes=0x808d260, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(117).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
alltoallw_zeros
8
./coll
fail
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff350d9538, rcnts=0x1ed121b0, rdispls=0x1ed12210, rtypes=0x1ed100a0, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(108).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Alltoallw: Other MPI error, error stack:
PMPI_Alltoallw(521)...............: MPI_Alltoallw(sbuf=MPI_IN_PLACE, scnts=(nil), sdispls=(nil), stypes=(nil), rbuf=0x7fff23653c48, rcnts=0xf4da1e0, rdispls=0xf4da240, rtypes=0xf4d80d0, MPI_COMM_WORLD) failed
MPIR_Alltoallw_impl(353)..........:
MPIR_Alltoallw(319)...............:
MPIR_Alltoallw_intra(117).........:
MPIC_Sendrecv_replace(271)........:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
[proxy:0:1@porthos] handle_pmi_response (./pm/pmiserv/pmip_cb.c:408): assert (!closed) failed
[proxy:0:1@porthos] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:895): unable to handle PMI response
[proxy:0:1@porthos] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:1@porthos] main (./pm/pmiserv/pmip.c:264): demux engine error waiting for event
[mpiexec@aramis] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:70): assert (!(pollfds[i].revents & ~POLLIN & ~POLLOUT & ~POLLHUP)) failed
[mpiexec@aramis] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:84): error waiting for event
[mpiexec@aramis] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:18): bootstrap device returned error waiting for completion
[mpiexec@aramis] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:325): bootstrap server returned error waiting for completion
[mpiexec@aramis] main (./ui/mpich/mpiexec.c:293): process manager error waiting for completion
allgather2
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff1dfdbc68, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff9435c838, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allgather3
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff550033c8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fffd1b89df8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allgatherv2
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff3a5680f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff030593d8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allgatherv3
10
./coll
fail
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff5dc3fd88, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in MPI_Allreduce: Other MPI error, error stack:
MPI_Allreduce(826)................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x7fff99fa96f8, count=1, MPI_INT, MPI_LAND, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(684)..........:
MPIR_Allreduce_intra(192).........:
allreduce_intra_or_coll_fn(103)...:
MPIR_Allreduce_intra(362).........:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
allgatherv4
4
./coll
fail
Fatal error in PMPI_Barrier: Other MPI error, error stack:
PMPI_Barrier(362).................: MPI_Barrier(MPI_COMM_WORLD) failed
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(895):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 1:
Fatal error in PMPI_Barrier: Other MPI error, error stack:
PMPI_Barrier(362).................: MPI_Barrier(MPI_COMM_WORLD) failed
MPIR_Barrier_impl(241)............:
MPIR_Barrier_or_coll_fn(112)......:
MPIR_Barrier_intra(79)............:
MPIC_Sendrecv(186)................:
MPIC_Wait(534)....................:
MPIDI_CH3I_Progress(184)..........:
MPID_nem_mpich2_blocking_recv(880):
MPID_nem_tcp_connpoll(1746).......: Communication error with rank 0:
APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
bcasttest
4
./coll