Great patch. It's solved. <br>
<br>
Another problem is still -binding rr. -binding rr looks still filling
out 1st node's all slots and then 2nd node's. Its binding looks the same with
cpu on my case. Do I need some trick on hostfile to reach an effect of
OpenMPI's -bynode or MVAPICH2's scatter? <br>
<br>tma@freims:~/sw/sw/imb/src$ mpiexec -n 25 -binding rr -f ~/host_mpich hostname<br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a><br><br>tma@freims:~/sw/sw/imb/src$ mpiexec -n 25 -binding cpu -f ~/host_mpich hostname<br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br><a href="http://stremi-4.reims.grid5000.fr">stremi-4.reims.grid5000.fr</a><br>
<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a><br><br>
Appreciate your help. <br>
Teng<br><br><div class="gmail_quote">On Tue, Aug 2, 2011 at 4:25 PM, Darius Buntinas <span dir="ltr"><<a href="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
If the DNS server is being overloaded, the following patch should help. Again, apply the patch, make clean ; make ; make install, rebuild IMB and see if it fails.<br>
<br>
Thanks,<br>
<font color="#888888">-d<br>
<br>
</font><br><br>
<br>
On Aug 2, 2011, at 2:35 PM, teng ma wrote:<br>
<br>
> 408 works this time. But when scale to 768, the same error comes out<br>
><br>
> [29] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [29] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [29] MPIR_Init_thread(388)..............:<br>
> [29] MPID_Init(139).....................: channel initialization failed<br>
> [29] MPIDI_CH3_Init(38).................:<br>
> [29] MPID_nem_init(234).................:<br>
> [29] MPID_nem_tcp_init(99)..............:<br>
> [29] MPID_nem_tcp_get_business_card(325):<br>
> [29] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [27] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [45] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [45] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [45] MPIR_Init_thread(388)..............:<br>
> [45] MPID_Init(139).....................: channel initialization failed<br>
> [45] MPIDI_CH3_Init(38).................:<br>
> [45] MPID_nem_init(234).................:<br>
> [45] MPID_nem_tcp_init(99)..............:<br>
> [45] MPID_nem_tcp_get_business_card(325):<br>
> [45] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [47] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [47] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [47] MPIR_Init_thread(388)..............:<br>
> [47] MPID_Init(139).....................: channel initialization failed<br>
> [47] MPIDI_CH3_Init(38).................:<br>
> [47] MPID_nem_init(234).................:<br>
> [47] MPID_nem_tcp_init(99)..............:<br>
> [47] MPID_nem_tcp_get_business_card(325):<br>
> [47] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [41] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [46] ifname="<a href="http://stremi-35.reims.grid5000.fr" target="_blank">stremi-35.reims.grid5000.fr</a>"<br>
> [41] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [41] MPIR_Init_thread(388)..............:<br>
> [41] MPID_Init(139).....................: channel initialization failed<br>
> [41] MPIDI_CH3_Init(38).................:<br>
> [41] MPID_nem_init(234).................:<br>
> [41] MPID_nem_tcp_init(99)..............:<br>
> [41] MPID_nem_tcp_get_business_card(325):<br>
> [41] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [46] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [46] MPIR_Init_thread(388)..............:<br>
> [46] MPID_Init(139).....................: channel initialization failed<br>
> [46] MPIDI_CH3_Init(38).................:<br>
> [46] MPID_nem_init(234).................:<br>
> [46] MPID_nem_tcp_init(99)..............:<br>
> [46] MPID_nem_tcp_get_business_card(325):<br>
> [46] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [27] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [27] MPIR_Init_thread(388)..............:<br>
> [27] MPID_Init(139).....................: channel initialization failed<br>
> [27] MPIDI_CH3_Init(38).................:<br>
> [27] MPID_nem_init(234).................:<br>
> [27] MPID_nem_tcp_init(99)..............:<br>
> [27] MPID_nem_tcp_get_business_card(325):<br>
> [27] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [120] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [122] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [123] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [127] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [134] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [120] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [120] MPIR_Init_thread(388)..............:<br>
> [120] MPID_Init(139).....................: channel initialization failed<br>
> [120] MPIDI_CH3_Init(38).................:<br>
> [120] MPID_nem_init(234).................:<br>
> [120] MPID_nem_tcp_init(99)..............:<br>
> [120] MPID_nem_tcp_get_business_card(325):<br>
> [120] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [122] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [122] MPIR_Init_thread(388)..............:<br>
> [122] MPID_Init(139).....................: channel initialization failed<br>
> [122] MPIDI_CH3_Init(38).................:<br>
> [122] MPID_nem_init(234).................:<br>
> [122] MPID_nem_tcp_init(99)..............:<br>
> [122] MPID_nem_tcp_get_business_card(325):<br>
> [122] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [123] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [123] MPIR_Init_thread(388)..............:<br>
> [123] MPID_Init(139).....................: channel initialization failed<br>
> [123] MPIDI_CH3_Init(38).................:<br>
> [123] MPID_nem_init(234).................:<br>
> [123] MPID_nem_tcp_init(99)..............:<br>
> [123] MPID_nem_tcp_get_business_card(325):<br>
> [123] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [127] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [127] MPIR_Init_thread(388)..............:<br>
> [127] MPID_Init(139).....................: channel initialization failed<br>
> [127] MPIDI_CH3_Init(38).................:<br>
> [127] MPID_nem_init(234).................:<br>
> [127] MPID_nem_tcp_init(99)..............:<br>
> [127] MPID_nem_tcp_get_business_card(325):<br>
> [127] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [134] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [134] MPIR_Init_thread(388)..............:<br>
> [134] MPID_Init(139).....................: channel initialization failed<br>
> [134] MPIDI_CH3_Init(38).................:<br>
> [134] MPID_nem_init(234).................:<br>
> [134] MPID_nem_tcp_init(99)..............:<br>
> [134] MPID_nem_tcp_get_business_card(325):<br>
> [134] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [125] ifname="<a href="http://stremi-22.reims.grid5000.fr" target="_blank">stremi-22.reims.grid5000.fr</a>"<br>
> [192] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [193] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [195] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [200] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [206] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [211] ifname="<a href="http://stremi-37.reims.grid5000.fr" target="_blank">stremi-37.reims.grid5000.fr</a>"<br>
> [98] ifname="<a href="http://stremi-24.reims.grid5000.fr" target="_blank">stremi-24.reims.grid5000.fr</a>"<br>
> [192] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [192] MPIR_Init_thread(388)..............:<br>
> [192] MPID_Init(139).....................: channel initialization failed<br>
> [192] MPIDI_CH3_Init(38).................:<br>
> [192] MPID_nem_init(234).................:<br>
> [192] MPID_nem_tcp_init(99)..............:<br>
> [192] MPID_nem_tcp_get_business_card(325):<br>
> [192] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [100] ifname="<a href="http://stremi-24.reims.grid5000.fr" target="_blank">stremi-24.reims.grid5000.fr</a>"<br>
> [195] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [195] MPIR_Init_thread(388)..............:<br>
> [195] MPID_Init(139).....................: channel initialization failed<br>
> [195] MPIDI_CH3_Init(38).................:<br>
> [195] MPID_nem_init(234).................:<br>
> [195] MPID_nem_tcp_init(99)..............:<br>
> [195] MPID_nem_tcp_get_business_card(325):<br>
> [195] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [110] ifname="<a href="http://stremi-24.reims.grid5000.fr" target="_blank">stremi-24.reims.grid5000.fr</a>"<br>
> [200] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [200] MPIR_Init_thread(388)..............:<br>
> [200] MPID_Init(139).....................: channel initialization failed<br>
> [200] MPIDI_CH3_Init(38).................:<br>
> [200] MPID_nem_init(234).................:<br>
> [200] MPID_nem_tcp_init(99)..............:<br>
> [200] MPID_nem_tcp_get_business_card(325):<br>
> [200] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [114] ifname="<a href="http://stremi-24.reims.grid5000.fr" target="_blank">stremi-24.reims.grid5000.fr</a>"<br>
> [211] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [211] MPIR_Init_thread(388)..............:<br>
> [211] MPID_Init(139).....................: channel initialization failed<br>
> [211] MPIDI_CH3_Init(38).................:<br>
> [211] MPID_nem_init(234).................:<br>
> [211] MPID_nem_tcp_init(99)..............:<br>
> [211] MPID_nem_tcp_get_business_card(325):<br>
> [211] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [98] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [98] MPIR_Init_thread(388)..............:<br>
> [98] MPID_Init(139).....................: channel initialization failed<br>
> [98] MPIDI_CH3_Init(38).................:<br>
> [98] MPID_nem_init(234).................:<br>
> [98] MPID_nem_tcp_init(99)..............:<br>
> [98] MPID_nem_tcp_get_business_card(325):<br>
> [98] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [193] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [193] MPIR_Init_thread(388)..............:<br>
> [193] MPID_Init(139).....................: channel initialization failed<br>
> [193] MPIDI_CH3_Init(38).................:<br>
> [193] MPID_nem_init(234).................:<br>
> [193] MPID_nem_tcp_init(99)..............:<br>
> [193] MPID_nem_tcp_get_business_card(325):<br>
> [193] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [100] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [100] MPIR_Init_thread(388)..............:<br>
> [100] MPID_Init(139).....................: channel initialization failed<br>
> [100] MPIDI_CH3_Init(38).................:<br>
> [100] MPID_nem_init(234).................:<br>
> [100] MPID_nem_tcp_init(99)..............:<br>
> [100] MPID_nem_tcp_get_business_card(325):<br>
> [100] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [206] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [206] MPIR_Init_thread(388)..............:<br>
> [206] MPID_Init(139).....................: channel initialization failed<br>
> [206] MPIDI_CH3_Init(38).................:<br>
> [206] MPID_nem_init(234).................:<br>
> [206] MPID_nem_tcp_init(99)..............:<br>
> [206] MPID_nem_tcp_get_business_card(325):<br>
> [206] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [110] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [110] MPIR_Init_thread(388)..............:<br>
> [110] MPID_Init(139).....................: channel initialization failed<br>
> [110] MPIDI_CH3_Init(38).................:<br>
> [110] MPID_nem_init(234).................:<br>
> [110] MPID_nem_tcp_init(99)..............:<br>
> [110] MPID_nem_tcp_get_business_card(325):<br>
> [110] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [114] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [114] MPIR_Init_thread(388)..............:<br>
> [114] MPID_Init(139).....................: channel initialization failed<br>
> [114] MPIDI_CH3_Init(38).................:<br>
> [114] MPID_nem_init(234).................:<br>
> [114] MPID_nem_tcp_init(99)..............:<br>
> [114] MPID_nem_tcp_get_business_card(325):<br>
> [114] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [697] ifname="<a href="http://stremi-5.reims.grid5000.fr" target="_blank">stremi-5.reims.grid5000.fr</a>"<br>
> [717] ifname="<a href="http://stremi-5.reims.grid5000.fr" target="_blank">stremi-5.reims.grid5000.fr</a>"<br>
> [504] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [718] ifname="<a href="http://stremi-5.reims.grid5000.fr" target="_blank">stremi-5.reims.grid5000.fr</a>"<br>
> [524] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [711] ifname="<a href="http://stremi-5.reims.grid5000.fr" target="_blank">stremi-5.reims.grid5000.fr</a>"<br>
> [524] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [524] MPIR_Init_thread(388)..............:<br>
> [524] MPID_Init(139).....................: channel initialization failed<br>
> [524] MPIDI_CH3_Init(38).................:<br>
> [524] MPID_nem_init(234).................:<br>
> [524] MPID_nem_tcp_init(99)..............:<br>
> [524] MPID_nem_tcp_get_business_card(325):<br>
> [524] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [697] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [697] MPIR_Init_thread(388)..............:<br>
> [697] MPID_Init(139).....................: channel initialization failed<br>
> [697] MPIDI_CH3_Init(38).................:<br>
> [697] MPID_nem_init(234).................:<br>
> [697] MPID_nem_tcp_init(99)..............:<br>
> [697] MPID_nem_tcp_get_business_card(325):<br>
> [697] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [508] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [711] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [711] MPIR_Init_thread(388)..............:<br>
> [711] MPID_Init(139).....................: channel initialization failed<br>
> [711] MPIDI_CH3_Init(38).................:<br>
> [711] MPID_nem_init(234).................:<br>
> [711] MPID_nem_tcp_init(99)..............:<br>
> [711] MPID_nem_tcp_get_business_card(325):<br>
> [711] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [504] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [504] MPIR_Init_thread(388)..............:<br>
> [504] MPID_Init(139).....................: channel initialization failed<br>
> [504] MPIDI_CH3_Init(38).................:<br>
> [504] MPID_nem_init(234).................:<br>
> [504] MPID_nem_tcp_init(99)..............:<br>
> [504] MPID_nem_tcp_get_business_card(325):<br>
> [504] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [717] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [717] MPIR_Init_thread(388)..............:<br>
> [717] MPID_Init(139).....................: channel initialization failed<br>
> [717] MPIDI_CH3_Init(38).................:<br>
> [717] MPID_nem_init(234).................:<br>
> [717] MPID_nem_tcp_init(99)..............:<br>
> [717] MPID_nem_tcp_get_business_card(325):<br>
> [717] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [508] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [508] MPIR_Init_thread(388)..............:<br>
> [508] MPID_Init(139).....................: channel initialization failed<br>
> [508] MPIDI_CH3_Init(38).................:<br>
> [508] MPID_nem_init(234).................:<br>
> [508] MPID_nem_tcp_init(99)..............:<br>
> [508] MPID_nem_tcp_get_business_card(325):<br>
> [508] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [718] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [718] MPIR_Init_thread(388)..............:<br>
> [718] MPID_Init(139).....................: channel initialization failed<br>
> [718] MPIDI_CH3_Init(38).................:<br>
> [718] MPID_nem_init(234).................:<br>
> [718] MPID_nem_tcp_init(99)..............:<br>
> [718] MPID_nem_tcp_get_business_card(325):<br>
> [718] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [512] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [507] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [444] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [509] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [511] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [515] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [518] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [523] ifname="<a href="http://stremi-43.reims.grid5000.fr" target="_blank">stremi-43.reims.grid5000.fr</a>"<br>
> [507] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [507] MPIR_Init_thread(388)..............:<br>
> [507] MPID_Init(139).....................: channel initialization failed<br>
> [507] MPIDI_CH3_Init(38).................:<br>
> [507] MPID_nem_init(234).................:<br>
> [507] MPID_nem_tcp_init(99)..............:<br>
> [507] MPID_nem_tcp_get_business_card(325):<br>
> [507] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [509] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [509] MPIR_Init_thread(388)..............:<br>
> [509] MPID_Init(139).....................: channel initialization failed<br>
> [509] MPIDI_CH3_Init(38).................:<br>
> [509] MPID_nem_init(234).................:<br>
> [509] MPID_nem_tcp_init(99)..............:<br>
> [509] MPID_nem_tcp_get_business_card(325):<br>
> [509] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [511] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [511] MPIR_Init_thread(388)..............:<br>
> [511] MPID_Init(139).....................: channel initialization failed<br>
> [511] MPIDI_CH3_Init(38).................:<br>
> [511] MPID_nem_init(234).................:<br>
> [511] MPID_nem_tcp_init(99)..............:<br>
> [511] MPID_nem_tcp_get_business_card(325):<br>
> [511] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [512] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [512] MPIR_Init_thread(388)..............:<br>
> [512] MPID_Init(139).....................: channel initialization failed<br>
> [512] MPIDI_CH3_Init(38).................:<br>
> [512] MPID_nem_init(234).................:<br>
> [512] MPID_nem_tcp_init(99)..............:<br>
> [512] MPID_nem_tcp_get_business_card(325):<br>
> [512] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [515] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [515] MPIR_Init_thread(388)..............:<br>
> [515] MPID_Init(139).....................: channel initialization failed<br>
> [515] MPIDI_CH3_Init(38).................:<br>
> [515] MPID_nem_init(234).................:<br>
> [515] MPID_nem_tcp_init(99)..............:<br>
> [515] MPID_nem_tcp_get_business_card(325):<br>
> [515] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [443] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [523] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [523] MPIR_Init_thread(388)..............:<br>
> [523] MPID_Init(139).....................: channel initialization failed<br>
> [523] MPIDI_CH3_Init(38).................:<br>
> [523] MPID_nem_init(234).................:<br>
> [523] MPID_nem_tcp_init(99)..............:<br>
> [523] MPID_nem_tcp_get_business_card(325):<br>
> [523] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [443] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [443] MPIR_Init_thread(388)..............:<br>
> [443] MPID_Init(139).....................: channel initialization failed<br>
> [443] MPIDI_CH3_Init(38).................:<br>
> [443] MPID_nem_init(234).................:<br>
> [443] MPID_nem_tcp_init(99)..............:<br>
> [443] MPID_nem_tcp_get_business_card(325):<br>
> [443] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [518] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [518] MPIR_Init_thread(388)..............:<br>
> [518] MPID_Init(139).....................: channel initialization failed<br>
> [518] MPIDI_CH3_Init(38).................:<br>
> [518] MPID_nem_init(234).................:<br>
> [518] MPID_nem_tcp_init(99)..............:<br>
> [518] MPID_nem_tcp_get_business_card(325):<br>
> [518] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [444] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [444] MPIR_Init_thread(388)..............:<br>
> [444] MPID_Init(139).....................: channel initialization failed<br>
> [444] MPIDI_CH3_Init(38).................:<br>
> [444] MPID_nem_init(234).................:<br>
> [444] MPID_nem_tcp_init(99)..............:<br>
> [444] MPID_nem_tcp_get_business_card(325):<br>
> [444] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [624] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [672] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [446] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [626] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [685] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [440] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [628] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [673] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [446] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [446] MPIR_Init_thread(388)..............:<br>
> [446] MPID_Init(139).....................: channel initialization failed<br>
> [446] MPIDI_CH3_Init(38).................:<br>
> [446] MPID_nem_init(234).................:<br>
> [446] MPID_nem_tcp_init(99)..............:<br>
> [446] MPID_nem_tcp_get_business_card(325):<br>
> [446] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [630] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [682] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [440] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [440] MPIR_Init_thread(388)..............:<br>
> [440] MPID_Init(139).....................: channel initialization failed<br>
> [440] MPIDI_CH3_Init(38).................:<br>
> [440] MPID_nem_init(234).................:<br>
> [440] MPID_nem_tcp_init(99)..............:<br>
> [440] MPID_nem_tcp_get_business_card(325):<br>
> [440] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [631] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [682] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [682] MPIR_Init_thread(388)..............:<br>
> [682] MPID_Init(139).....................: channel initialization failed<br>
> [682] MPIDI_CH3_Init(38).................:<br>
> [682] MPID_nem_init(234).................:<br>
> [682] MPID_nem_tcp_init(99)..............:<br>
> [682] MPID_nem_tcp_get_business_card(325):<br>
> [682] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [450] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [624] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [624] MPIR_Init_thread(388)..............:<br>
> [624] MPID_Init(139).....................: channel initialization failed<br>
> [624] MPIDI_CH3_Init(38).................:<br>
> [624] MPID_nem_init(234).................:<br>
> [624] MPID_nem_tcp_init(99)..............:<br>
> [624] MPID_nem_tcp_get_business_card(325):<br>
> [624] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [685] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [685] MPIR_Init_thread(388)..............:<br>
> [685] MPID_Init(139).....................: channel initialization failed<br>
> [685] MPIDI_CH3_Init(38).................:<br>
> [685] MPID_nem_init(234).................:<br>
> [685] MPID_nem_tcp_init(99)..............:<br>
> [685] MPID_nem_tcp_get_business_card(325):<br>
> [685] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [441] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [626] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [626] MPIR_Init_thread(388)..............:<br>
> [626] MPID_Init(139).....................: channel initialization failed<br>
> [626] MPIDI_CH3_Init(38).................:<br>
> [626] MPID_nem_init(234).................:<br>
> [626] MPID_nem_tcp_init(99)..............:<br>
> [626] MPID_nem_tcp_get_business_card(325):<br>
> [626] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [678] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [447] ifname="<a href="http://stremi-33.reims.grid5000.fr" target="_blank">stremi-33.reims.grid5000.fr</a>"<br>
> [628] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [628] MPIR_Init_thread(388)..............:<br>
> [628] MPID_Init(139).....................: channel initialization failed<br>
> [628] MPIDI_CH3_Init(38).................:<br>
> [628] MPID_nem_init(234).................:<br>
> [628] MPID_nem_tcp_init(99)..............:<br>
> [628] MPID_nem_tcp_get_business_card(325):<br>
> [628] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [687] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [447] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [447] MPIR_Init_thread(388)..............:<br>
> [447] MPID_Init(139).....................: channel initialization failed<br>
> [447] MPIDI_CH3_Init(38).................:<br>
> [447] MPID_nem_init(234).................:<br>
> [447] MPID_nem_tcp_init(99)..............:<br>
> [447] MPID_nem_tcp_get_business_card(325):<br>
> [447] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [630] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [630] MPIR_Init_thread(388)..............:<br>
> [630] MPID_Init(139).....................: channel initialization failed<br>
> [630] MPIDI_CH3_Init(38).................:<br>
> [630] MPID_nem_init(234).................:<br>
> [630] MPID_nem_tcp_init(99)..............:<br>
> [630] MPID_nem_tcp_get_business_card(325):<br>
> [630] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [672] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [672] MPIR_Init_thread(388)..............:<br>
> [672] MPID_Init(139).....................: channel initialization failed<br>
> [672] MPIDI_CH3_Init(38).................:<br>
> [672] MPID_nem_init(234).................:<br>
> [672] MPID_nem_tcp_init(99)..............:<br>
> [672] MPID_nem_tcp_get_business_card(325):<br>
> [672] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [450] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [450] MPIR_Init_thread(388)..............:<br>
> [450] MPID_Init(139).....................: channel initialization failed<br>
> [450] MPIDI_CH3_Init(38).................:<br>
> [450] MPID_nem_init(234).................:<br>
> [450] MPID_nem_tcp_init(99)..............:<br>
> [450] MPID_nem_tcp_get_business_card(325):<br>
> [450] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [627] ifname="<a href="http://stremi-27.reims.grid5000.fr" target="_blank">stremi-27.reims.grid5000.fr</a>"<br>
> [673] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [673] MPIR_Init_thread(388)..............:<br>
> [673] MPID_Init(139).....................: channel initialization failed<br>
> [673] MPIDI_CH3_Init(38).................:<br>
> [673] MPID_nem_init(234).................:<br>
> [673] MPID_nem_tcp_init(99)..............:<br>
> [673] MPID_nem_tcp_get_business_card(325):<br>
> [673] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [627] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [627] MPIR_Init_thread(388)..............:<br>
> [627] MPID_Init(139).....................: channel initialization failed<br>
> [627] MPIDI_CH3_Init(38).................:<br>
> [627] MPID_nem_init(234).................:<br>
> [627] MPID_nem_tcp_init(99)..............:<br>
> [627] MPID_nem_tcp_get_business_card(325):<br>
> [627] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [687] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [687] MPIR_Init_thread(388)..............:<br>
> [687] MPID_Init(139).....................: channel initialization failed<br>
> [687] MPIDI_CH3_Init(38).................:<br>
> [687] MPID_nem_init(234).................:<br>
> [687] MPID_nem_tcp_init(99)..............:<br>
> [687] MPID_nem_tcp_get_business_card(325):<br>
> [687] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [631] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [631] MPIR_Init_thread(388)..............:<br>
> [631] MPID_Init(139).....................: channel initialization failed<br>
> [631] MPIDI_CH3_Init(38).................:<br>
> [631] MPID_nem_init(234).................:<br>
> [631] MPID_nem_tcp_init(99)..............:<br>
> [631] MPID_nem_tcp_get_business_card(325):<br>
> [631] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [683] ifname="<a href="http://stremi-25.reims.grid5000.fr" target="_blank">stremi-25.reims.grid5000.fr</a>"<br>
> [678] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [678] MPIR_Init_thread(388)..............:<br>
> [678] MPID_Init(139).....................: channel initialization failed<br>
> [678] MPIDI_CH3_Init(38).................:<br>
> [678] MPID_nem_init(234).................:<br>
> [678] MPID_nem_tcp_init(99)..............:<br>
> [678] MPID_nem_tcp_get_business_card(325):<br>
> [678] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> [683] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> [683] MPIR_Init_thread(388)..............:<br>
> [683] MPID_Init(139).....................: channel initialization failed<br>
> [683] MPIDI_CH3_Init(38).................:<br>
> [683] MPID_nem_init(234).................:<br>
> [683] MPID_nem_tcp_init(99)..............:<br>
> [683] MPID_nem_tcp_get_business_card(325):<br>
> [683] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
><br>
><br>
> On Tue, Aug 2, 2011 at 3:14 PM, Darius Buntinas <<a href="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</a>> wrote:<br>
> OK, can you apply the attached patch, rebuild mpich2 and IMB, then re-run the test with the options that gave the errors?<br>
><br>
> The patch should give us more info on the error.<br>
><br>
> To apply the patch, do this from the mpich2 source directory:<br>
> patch -p0 < dbg.diff<br>
><br>
> Then to rebuild mpich2:<br>
> make clean<br>
> make<br>
> make install<br>
><br>
> Then, after rebuilding IMB, re-run it like this:<br>
> mpiexec -l -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408<br>
><br>
> Thanks,<br>
> -d<br>
><br>
><br>
><br>
><br>
> On Aug 2, 2011, at 1:23 PM, teng ma wrote:<br>
><br>
> > tma@freims:~$ mpiexec -l -n 2 -binding cpu -f ~/host_mpich env<br>
> > [0] SHELL=/bin/bash<br>
> > [0] SSH_CLIENT=192.168.159.239 59246 22<br>
> > [0] LC_ALL=en_US.UTF-8<br>
> > [0] USER=tma<br>
> > [0] MAIL=/var/mail/tma<br>
> > [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> > [0] PWD=/home/tma<br>
> > [0] LANG=en_US.UTF-8<br>
> > [0] SHLVL=1<br>
> > [0] HOME=/home/tma<br>
> > [0] LOGNAME=tma<br>
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> > [0] _=/home/tma/opt/mpi/bin/mpiexec<br>
> > [0] TERM=xterm<br>
> > [0] OLDPWD=/home/tma/opt/mpi<br>
> > [0] SSH_TTY=/dev/pts/26<br>
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> > [0] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> > [0] PMI_RANK=0<br>
> > [0] PMI_FD=6<br>
> > [0] PMI_SIZE=2<br>
> > [1] SHELL=/bin/bash<br>
> > [1] SSH_CLIENT=192.168.159.239 59246 22<br>
> > [1] LC_ALL=en_US.UTF-8<br>
> > [1] USER=tma<br>
> > [1] MAIL=/var/mail/tma<br>
> > [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> > [1] PWD=/home/tma<br>
> > [1] LANG=en_US.UTF-8<br>
> > [1] SHLVL=1<br>
> > [1] HOME=/home/tma<br>
> > [1] LOGNAME=tma<br>
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> > [1] _=/home/tma/opt/mpi/bin/mpiexec<br>
> > [1] TERM=xterm<br>
> > [1] OLDPWD=/home/tma/opt/mpi<br>
> > [1] SSH_TTY=/dev/pts/26<br>
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> > [1] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> > [1] PMI_RANK=1<br>
> > [1] PMI_FD=7<br>
> > [1] PMI_SIZE=2<br>
> ><br>
> ><br>
> > and<br>
> ><br>
> ><br>
> > tma@freims:~$ mpiexec -l -n 2 -f ~/host_mpich env<br>
> > [0] SHELL=/bin/bash<br>
> > [0] SSH_CLIENT=192.168.159.239 59246 22<br>
> > [0] LC_ALL=en_US.UTF-8<br>
> > [0] USER=tma<br>
> > [0] MAIL=/var/mail/tma<br>
> > [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> > [0] PWD=/home/tma<br>
> > [0] LANG=en_US.UTF-8<br>
> > [0] SHLVL=1<br>
> > [0] HOME=/home/tma<br>
> > [0] LOGNAME=tma<br>
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> > [0] _=/home/tma/opt/mpi/bin/mpiexec<br>
> > [0] TERM=xterm<br>
> > [0] OLDPWD=/home/tma/opt/mpi<br>
> > [0] SSH_TTY=/dev/pts/26<br>
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> > [0] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> > [0] PMI_RANK=0<br>
> > [0] PMI_FD=5<br>
> > [0] PMI_SIZE=2<br>
> > [1] SHELL=/bin/bash<br>
> > [1] SSH_CLIENT=192.168.159.239 59246 22<br>
> > [1] LC_ALL=en_US.UTF-8<br>
> > [1] USER=tma<br>
> > [1] MAIL=/var/mail/tma<br>
> > [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> > [1] PWD=/home/tma<br>
> > [1] LANG=en_US.UTF-8<br>
> > [1] SHLVL=1<br>
> > [1] HOME=/home/tma<br>
> > [1] LOGNAME=tma<br>
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> > [1] _=/home/tma/opt/mpi/bin/mpiexec<br>
> > [1] TERM=xterm<br>
> > [1] OLDPWD=/home/tma/opt/mpi<br>
> > [1] SSH_TTY=/dev/pts/26<br>
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> > [1] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> > [1] PMI_RANK=1<br>
> > [1] PMI_FD=6<br>
> > [1] PMI_SIZE=2<br>
> ><br>
> ><br>
> ><br>
> > On Tue, Aug 2, 2011 at 1:49 PM, Darius Buntinas <<a href="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > Can you send us the output of the following?<br>
> ><br>
> > mpiexec -l -n 2 -binding cpu -f ~/host_mpich env<br>
> > and<br>
> > mpiexec -l -n 2 -f ~/host_mpich env<br>
> ><br>
> > Thanks,<br>
> > -d<br>
> ><br>
> > On Aug 2, 2011, at 12:18 PM, teng ma wrote:<br>
> ><br>
> > > If -binding is removed, it's no problem to scale to 768 processes. (32 nodes, 24 core /node). if without binding parameter, what kind of binding strategy mpich2 will use? ( fill out all slots of one nodes, and then another node, or round robin along nodes?)<br>
> > ><br>
> > > Thanks<br>
> > > Teng<br>
> > ><br>
> > > On Tue, Aug 2, 2011 at 1:14 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>> wrote:<br>
> > ><br>
> > > Please keep mpich-discuss cc'ed. The below error doesn't seem to be a binding issue. Did you try removing the -binding option to see if it works without that?<br>
> > ><br>
> > ><br>
> > > On 08/02/2011 12:12 PM, teng ma wrote:<br>
> > > thanks for the answer. I met another issue with hydra binding. When<br>
> > > processes launched exceed 408, it throws error like following:<br>
> > ><br>
> > ><br>
> > > I run it like<br>
> > > mpiexec -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > > MPIR_Init_thread(388)..............:<br>
> > > MPID_Init(139).....................: channel initialization failed<br>
> > > MPIDI_CH3_Init(38).................:<br>
> > > MPID_nem_init(234).................:<br>
> > > MPID_nem_tcp_init(99)..............:<br>
> > > MPID_nem_tcp_get_business_card(325):<br>
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > ><br>
> > ><br>
> > > When processes is less than 407, -binding cpu/rr looks good. If I<br>
> > > remove -binding cpu/rr, just with -f ~/host_mpich, it's still ok no<br>
> > > matter how many processes. My host_mpich is like:<br>
> > ><br>
> > > <a href="http://stremi-7.reims.grid5000.fr:24" target="_blank">stremi-7.reims.grid5000.fr:24</a> <<a href="http://stremi-7.reims.grid5000.fr:24" target="_blank">http://stremi-7.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-35.reims.grid5000.fr:24" target="_blank">stremi-35.reims.grid5000.fr:24</a> <<a href="http://stremi-35.reims.grid5000.fr:24" target="_blank">http://stremi-35.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-28.reims.grid5000.fr:24" target="_blank">stremi-28.reims.grid5000.fr:24</a> <<a href="http://stremi-28.reims.grid5000.fr:24" target="_blank">http://stremi-28.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-38.reims.grid5000.fr:24" target="_blank">stremi-38.reims.grid5000.fr:24</a> <<a href="http://stremi-38.reims.grid5000.fr:24" target="_blank">http://stremi-38.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-32.reims.grid5000.fr:24" target="_blank">stremi-32.reims.grid5000.fr:24</a> <<a href="http://stremi-32.reims.grid5000.fr:24" target="_blank">http://stremi-32.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-26.reims.grid5000.fr:24" target="_blank">stremi-26.reims.grid5000.fr:24</a> <<a href="http://stremi-26.reims.grid5000.fr:24" target="_blank">http://stremi-26.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-22.reims.grid5000.fr:24" target="_blank">stremi-22.reims.grid5000.fr:24</a> <<a href="http://stremi-22.reims.grid5000.fr:24" target="_blank">http://stremi-22.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-43.reims.grid5000.fr:24" target="_blank">stremi-43.reims.grid5000.fr:24</a> <<a href="http://stremi-43.reims.grid5000.fr:24" target="_blank">http://stremi-43.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-30.reims.grid5000.fr:24" target="_blank">stremi-30.reims.grid5000.fr:24</a> <<a href="http://stremi-30.reims.grid5000.fr:24" target="_blank">http://stremi-30.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-41.reims.grid5000.fr:24" target="_blank">stremi-41.reims.grid5000.fr:24</a> <<a href="http://stremi-41.reims.grid5000.fr:24" target="_blank">http://stremi-41.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-4.reims.grid5000.fr:24" target="_blank">stremi-4.reims.grid5000.fr:24</a> <<a href="http://stremi-4.reims.grid5000.fr:24" target="_blank">http://stremi-4.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-34.reims.grid5000.fr:24" target="_blank">stremi-34.reims.grid5000.fr:24</a> <<a href="http://stremi-34.reims.grid5000.fr:24" target="_blank">http://stremi-34.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-24.reims.grid5000.fr:24" target="_blank">stremi-24.reims.grid5000.fr:24</a> <<a href="http://stremi-24.reims.grid5000.fr:24" target="_blank">http://stremi-24.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-23.reims.grid5000.fr:24" target="_blank">stremi-23.reims.grid5000.fr:24</a> <<a href="http://stremi-23.reims.grid5000.fr:24" target="_blank">http://stremi-23.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-20.reims.grid5000.fr:24" target="_blank">stremi-20.reims.grid5000.fr:24</a> <<a href="http://stremi-20.reims.grid5000.fr:24" target="_blank">http://stremi-20.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-36.reims.grid5000.fr:24" target="_blank">stremi-36.reims.grid5000.fr:24</a> <<a href="http://stremi-36.reims.grid5000.fr:24" target="_blank">http://stremi-36.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-29.reims.grid5000.fr:24" target="_blank">stremi-29.reims.grid5000.fr:24</a> <<a href="http://stremi-29.reims.grid5000.fr:24" target="_blank">http://stremi-29.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-19.reims.grid5000.fr:24" target="_blank">stremi-19.reims.grid5000.fr:24</a> <<a href="http://stremi-19.reims.grid5000.fr:24" target="_blank">http://stremi-19.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-42.reims.grid5000.fr:24" target="_blank">stremi-42.reims.grid5000.fr:24</a> <<a href="http://stremi-42.reims.grid5000.fr:24" target="_blank">http://stremi-42.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-39.reims.grid5000.fr:24" target="_blank">stremi-39.reims.grid5000.fr:24</a> <<a href="http://stremi-39.reims.grid5000.fr:24" target="_blank">http://stremi-39.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-27.reims.grid5000.fr:24" target="_blank">stremi-27.reims.grid5000.fr:24</a> <<a href="http://stremi-27.reims.grid5000.fr:24" target="_blank">http://stremi-27.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-44.reims.grid5000.fr:24" target="_blank">stremi-44.reims.grid5000.fr:24</a> <<a href="http://stremi-44.reims.grid5000.fr:24" target="_blank">http://stremi-44.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-37.reims.grid5000.fr:24" target="_blank">stremi-37.reims.grid5000.fr:24</a> <<a href="http://stremi-37.reims.grid5000.fr:24" target="_blank">http://stremi-37.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-31.reims.grid5000.fr:24" target="_blank">stremi-31.reims.grid5000.fr:24</a> <<a href="http://stremi-31.reims.grid5000.fr:24" target="_blank">http://stremi-31.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-6.reims.grid5000.fr:24" target="_blank">stremi-6.reims.grid5000.fr:24</a> <<a href="http://stremi-6.reims.grid5000.fr:24" target="_blank">http://stremi-6.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-33.reims.grid5000.fr:24" target="_blank">stremi-33.reims.grid5000.fr:24</a> <<a href="http://stremi-33.reims.grid5000.fr:24" target="_blank">http://stremi-33.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-3.reims.grid5000.fr:24" target="_blank">stremi-3.reims.grid5000.fr:24</a> <<a href="http://stremi-3.reims.grid5000.fr:24" target="_blank">http://stremi-3.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-2.reims.grid5000.fr:24" target="_blank">stremi-2.reims.grid5000.fr:24</a> <<a href="http://stremi-2.reims.grid5000.fr:24" target="_blank">http://stremi-2.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-40.reims.grid5000.fr:24" target="_blank">stremi-40.reims.grid5000.fr:24</a> <<a href="http://stremi-40.reims.grid5000.fr:24" target="_blank">http://stremi-40.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-21.reims.grid5000.fr:24" target="_blank">stremi-21.reims.grid5000.fr:24</a> <<a href="http://stremi-21.reims.grid5000.fr:24" target="_blank">http://stremi-21.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-5.reims.grid5000.fr:24" target="_blank">stremi-5.reims.grid5000.fr:24</a> <<a href="http://stremi-5.reims.grid5000.fr:24" target="_blank">http://stremi-5.reims.grid5000.fr:24</a>><br>
> > > <a href="http://stremi-25.reims.grid5000.fr:24" target="_blank">stremi-25.reims.grid5000.fr:24</a> <<a href="http://stremi-25.reims.grid5000.fr:24" target="_blank">http://stremi-25.reims.grid5000.fr:24</a>><br>
> > ><br>
> > ><br>
> > > The configure of mpich2 is just default configure.<br>
> > ><br>
> > > Thanks<br>
> > > Teng<br>
> > ><br>
> > > On Tue, Aug 2, 2011 at 12:43 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a><br>
> > > <mailto:<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>>> wrote:<br>
> > ><br>
> > ><br>
> > > mpiexec -binding rr<br>
> > ><br>
> > > -- Pavan<br>
> > ><br>
> > ><br>
> > > On 08/02/2011 11:35 AM, teng ma wrote:<br>
> > ><br>
> > > If I want to do a process-core binding like MVAPICH2's scatter way:<br>
> > > assign MPI ranks by nodes in host file, e.g.<br>
> > > host1<br>
> > > host2<br>
> > > host3<br>
> > ><br>
> > > rank 0 host 1's core 0<br>
> > > rank 1 host 2's core 0<br>
> > > rank 2 host 3's core 0<br>
> > > rank 3 host 1's core 1<br>
> > > rank 4 host 2's core 1<br>
> > > rank 5 host 3's core 1<br>
> > ><br>
> > > Is there any easy method in mpich2-1.4 to achieve this binding?<br>
> > ><br>
> > > Teng Ma<br>
> > ><br>
> > ><br>
> > ><br>
> > > _________________________________________________<br>
> > > mpich-discuss mailing list<br>
> > > <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a> <mailto:<a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a>><br>
> > ><br>
> > > <a href="https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss</a><br>
> > > <<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a>><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Pavan Balaji<br>
> > > <a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a> <<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/%7Ebalaji</a>><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Pavan Balaji<br>
> > > <a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
> > ><br>
> > > _______________________________________________<br>
> > > mpich-discuss mailing list<br>
> > > <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
> > > <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
> ><br>
> > _______________________________________________<br>
> > mpich-discuss mailing list<br>
> > <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
> > <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
> ><br>
><br>
><br>
><br>
<br>
<br></blockquote></div><br>