408 works this time. But when scale to 768, the same error comes out<br><br>[29] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>[29] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[29] MPIR_Init_thread(388)..............: <br>[29] MPID_Init(139).....................: channel initialization failed<br>[29] MPIDI_CH3_Init(38).................: <br>[29] MPID_nem_init(234).................: <br>[29] MPID_nem_tcp_init(99)..............: <br>
[29] MPID_nem_tcp_get_business_card(325): <br>[29] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[27] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>
[45] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>[45] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[45] MPIR_Init_thread(388)..............: <br>
[45] MPID_Init(139).....................: channel initialization failed<br>[45] MPIDI_CH3_Init(38).................: <br>[45] MPID_nem_init(234).................: <br>[45] MPID_nem_tcp_init(99)..............: <br>[45] MPID_nem_tcp_get_business_card(325): <br>
[45] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[47] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>[47] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[47] MPIR_Init_thread(388)..............: <br>[47] MPID_Init(139).....................: channel initialization failed<br>[47] MPIDI_CH3_Init(38).................: <br>[47] MPID_nem_init(234).................: <br>[47] MPID_nem_tcp_init(99)..............: <br>
[47] MPID_nem_tcp_get_business_card(325): <br>[47] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[41] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>
[46] ifname="<a href="http://stremi-35.reims.grid5000.fr">stremi-35.reims.grid5000.fr</a>"<br>[41] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[41] MPIR_Init_thread(388)..............: <br>
[41] MPID_Init(139).....................: channel initialization failed<br>[41] MPIDI_CH3_Init(38).................: <br>[41] MPID_nem_init(234).................: <br>[41] MPID_nem_tcp_init(99)..............: <br>[41] MPID_nem_tcp_get_business_card(325): <br>
[41] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[46] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[46] MPIR_Init_thread(388)..............: <br>[46] MPID_Init(139).....................: channel initialization failed<br>
[46] MPIDI_CH3_Init(38).................: <br>[46] MPID_nem_init(234).................: <br>[46] MPID_nem_tcp_init(99)..............: <br>[46] MPID_nem_tcp_get_business_card(325): <br>[46] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[27] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[27] MPIR_Init_thread(388)..............: <br>[27] MPID_Init(139).....................: channel initialization failed<br>[27] MPIDI_CH3_Init(38).................: <br>
[27] MPID_nem_init(234).................: <br>[27] MPID_nem_tcp_init(99)..............: <br>[27] MPID_nem_tcp_get_business_card(325): <br>[27] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[120] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>[122] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>[123] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>
[127] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>[134] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>[120] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[120] MPIR_Init_thread(388)..............: <br>[120] MPID_Init(139).....................: channel initialization failed<br>[120] MPIDI_CH3_Init(38).................: <br>[120] MPID_nem_init(234).................: <br>[120] MPID_nem_tcp_init(99)..............: <br>
[120] MPID_nem_tcp_get_business_card(325): <br>[120] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[122] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[122] MPIR_Init_thread(388)..............: <br>
[122] MPID_Init(139).....................: channel initialization failed<br>[122] MPIDI_CH3_Init(38).................: <br>[122] MPID_nem_init(234).................: <br>[122] MPID_nem_tcp_init(99)..............: <br>[122] MPID_nem_tcp_get_business_card(325): <br>
[122] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[123] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[123] MPIR_Init_thread(388)..............: <br>[123] MPID_Init(139).....................: channel initialization failed<br>
[123] MPIDI_CH3_Init(38).................: <br>[123] MPID_nem_init(234).................: <br>[123] MPID_nem_tcp_init(99)..............: <br>[123] MPID_nem_tcp_get_business_card(325): <br>[123] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[127] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[127] MPIR_Init_thread(388)..............: <br>[127] MPID_Init(139).....................: channel initialization failed<br>[127] MPIDI_CH3_Init(38).................: <br>
[127] MPID_nem_init(234).................: <br>[127] MPID_nem_tcp_init(99)..............: <br>[127] MPID_nem_tcp_get_business_card(325): <br>[127] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[134] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[134] MPIR_Init_thread(388)..............: <br>[134] MPID_Init(139).....................: channel initialization failed<br>[134] MPIDI_CH3_Init(38).................: <br>
[134] MPID_nem_init(234).................: <br>[134] MPID_nem_tcp_init(99)..............: <br>[134] MPID_nem_tcp_get_business_card(325): <br>[134] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[125] ifname="<a href="http://stremi-22.reims.grid5000.fr">stremi-22.reims.grid5000.fr</a>"<br>[192] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>[193] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>
[195] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>[200] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>[206] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>
[211] ifname="<a href="http://stremi-37.reims.grid5000.fr">stremi-37.reims.grid5000.fr</a>"<br>[98] ifname="<a href="http://stremi-24.reims.grid5000.fr">stremi-24.reims.grid5000.fr</a>"<br>[192] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[192] MPIR_Init_thread(388)..............: <br>[192] MPID_Init(139).....................: channel initialization failed<br>[192] MPIDI_CH3_Init(38).................: <br>[192] MPID_nem_init(234).................: <br>[192] MPID_nem_tcp_init(99)..............: <br>
[192] MPID_nem_tcp_get_business_card(325): <br>[192] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[100] ifname="<a href="http://stremi-24.reims.grid5000.fr">stremi-24.reims.grid5000.fr</a>"<br>
[195] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[195] MPIR_Init_thread(388)..............: <br>[195] MPID_Init(139).....................: channel initialization failed<br>[195] MPIDI_CH3_Init(38).................: <br>
[195] MPID_nem_init(234).................: <br>[195] MPID_nem_tcp_init(99)..............: <br>[195] MPID_nem_tcp_get_business_card(325): <br>[195] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[110] ifname="<a href="http://stremi-24.reims.grid5000.fr">stremi-24.reims.grid5000.fr</a>"<br>[200] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[200] MPIR_Init_thread(388)..............: <br>
[200] MPID_Init(139).....................: channel initialization failed<br>[200] MPIDI_CH3_Init(38).................: <br>[200] MPID_nem_init(234).................: <br>[200] MPID_nem_tcp_init(99)..............: <br>[200] MPID_nem_tcp_get_business_card(325): <br>
[200] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[114] ifname="<a href="http://stremi-24.reims.grid5000.fr">stremi-24.reims.grid5000.fr</a>"<br>[211] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[211] MPIR_Init_thread(388)..............: <br>[211] MPID_Init(139).....................: channel initialization failed<br>[211] MPIDI_CH3_Init(38).................: <br>[211] MPID_nem_init(234).................: <br>[211] MPID_nem_tcp_init(99)..............: <br>
[211] MPID_nem_tcp_get_business_card(325): <br>[211] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[98] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[98] MPIR_Init_thread(388)..............: <br>
[98] MPID_Init(139).....................: channel initialization failed<br>[98] MPIDI_CH3_Init(38).................: <br>[98] MPID_nem_init(234).................: <br>[98] MPID_nem_tcp_init(99)..............: <br>[98] MPID_nem_tcp_get_business_card(325): <br>
[98] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[193] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[193] MPIR_Init_thread(388)..............: <br>[193] MPID_Init(139).....................: channel initialization failed<br>
[193] MPIDI_CH3_Init(38).................: <br>[193] MPID_nem_init(234).................: <br>[193] MPID_nem_tcp_init(99)..............: <br>[193] MPID_nem_tcp_get_business_card(325): <br>[193] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[100] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[100] MPIR_Init_thread(388)..............: <br>[100] MPID_Init(139).....................: channel initialization failed<br>[100] MPIDI_CH3_Init(38).................: <br>
[100] MPID_nem_init(234).................: <br>[100] MPID_nem_tcp_init(99)..............: <br>[100] MPID_nem_tcp_get_business_card(325): <br>[100] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[206] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[206] MPIR_Init_thread(388)..............: <br>[206] MPID_Init(139).....................: channel initialization failed<br>[206] MPIDI_CH3_Init(38).................: <br>
[206] MPID_nem_init(234).................: <br>[206] MPID_nem_tcp_init(99)..............: <br>[206] MPID_nem_tcp_get_business_card(325): <br>[206] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[110] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[110] MPIR_Init_thread(388)..............: <br>[110] MPID_Init(139).....................: channel initialization failed<br>[110] MPIDI_CH3_Init(38).................: <br>
[110] MPID_nem_init(234).................: <br>[110] MPID_nem_tcp_init(99)..............: <br>[110] MPID_nem_tcp_get_business_card(325): <br>[110] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[114] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[114] MPIR_Init_thread(388)..............: <br>[114] MPID_Init(139).....................: channel initialization failed<br>[114] MPIDI_CH3_Init(38).................: <br>
[114] MPID_nem_init(234).................: <br>[114] MPID_nem_tcp_init(99)..............: <br>[114] MPID_nem_tcp_get_business_card(325): <br>[114] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[697] ifname="<a href="http://stremi-5.reims.grid5000.fr">stremi-5.reims.grid5000.fr</a>"<br>[717] ifname="<a href="http://stremi-5.reims.grid5000.fr">stremi-5.reims.grid5000.fr</a>"<br>[504] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>
[718] ifname="<a href="http://stremi-5.reims.grid5000.fr">stremi-5.reims.grid5000.fr</a>"<br>[524] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[711] ifname="<a href="http://stremi-5.reims.grid5000.fr">stremi-5.reims.grid5000.fr</a>"<br>
[524] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[524] MPIR_Init_thread(388)..............: <br>[524] MPID_Init(139).....................: channel initialization failed<br>[524] MPIDI_CH3_Init(38).................: <br>
[524] MPID_nem_init(234).................: <br>[524] MPID_nem_tcp_init(99)..............: <br>[524] MPID_nem_tcp_get_business_card(325): <br>[524] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[697] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[697] MPIR_Init_thread(388)..............: <br>[697] MPID_Init(139).....................: channel initialization failed<br>[697] MPIDI_CH3_Init(38).................: <br>
[697] MPID_nem_init(234).................: <br>[697] MPID_nem_tcp_init(99)..............: <br>[697] MPID_nem_tcp_get_business_card(325): <br>[697] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[508] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[711] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[711] MPIR_Init_thread(388)..............: <br>
[711] MPID_Init(139).....................: channel initialization failed<br>[711] MPIDI_CH3_Init(38).................: <br>[711] MPID_nem_init(234).................: <br>[711] MPID_nem_tcp_init(99)..............: <br>[711] MPID_nem_tcp_get_business_card(325): <br>
[711] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[504] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[504] MPIR_Init_thread(388)..............: <br>[504] MPID_Init(139).....................: channel initialization failed<br>
[504] MPIDI_CH3_Init(38).................: <br>[504] MPID_nem_init(234).................: <br>[504] MPID_nem_tcp_init(99)..............: <br>[504] MPID_nem_tcp_get_business_card(325): <br>[504] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[717] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[717] MPIR_Init_thread(388)..............: <br>[717] MPID_Init(139).....................: channel initialization failed<br>[717] MPIDI_CH3_Init(38).................: <br>
[717] MPID_nem_init(234).................: <br>[717] MPID_nem_tcp_init(99)..............: <br>[717] MPID_nem_tcp_get_business_card(325): <br>[717] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[508] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[508] MPIR_Init_thread(388)..............: <br>[508] MPID_Init(139).....................: channel initialization failed<br>[508] MPIDI_CH3_Init(38).................: <br>
[508] MPID_nem_init(234).................: <br>[508] MPID_nem_tcp_init(99)..............: <br>[508] MPID_nem_tcp_get_business_card(325): <br>[508] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[718] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[718] MPIR_Init_thread(388)..............: <br>[718] MPID_Init(139).....................: channel initialization failed<br>[718] MPIDI_CH3_Init(38).................: <br>
[718] MPID_nem_init(234).................: <br>[718] MPID_nem_tcp_init(99)..............: <br>[718] MPID_nem_tcp_get_business_card(325): <br>[718] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[512] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[507] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[444] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>
[509] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[511] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[515] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>
[518] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[523] ifname="<a href="http://stremi-43.reims.grid5000.fr">stremi-43.reims.grid5000.fr</a>"<br>[507] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[507] MPIR_Init_thread(388)..............: <br>[507] MPID_Init(139).....................: channel initialization failed<br>[507] MPIDI_CH3_Init(38).................: <br>[507] MPID_nem_init(234).................: <br>[507] MPID_nem_tcp_init(99)..............: <br>
[507] MPID_nem_tcp_get_business_card(325): <br>[507] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[509] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[509] MPIR_Init_thread(388)..............: <br>
[509] MPID_Init(139).....................: channel initialization failed<br>[509] MPIDI_CH3_Init(38).................: <br>[509] MPID_nem_init(234).................: <br>[509] MPID_nem_tcp_init(99)..............: <br>[509] MPID_nem_tcp_get_business_card(325): <br>
[509] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[511] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[511] MPIR_Init_thread(388)..............: <br>[511] MPID_Init(139).....................: channel initialization failed<br>
[511] MPIDI_CH3_Init(38).................: <br>[511] MPID_nem_init(234).................: <br>[511] MPID_nem_tcp_init(99)..............: <br>[511] MPID_nem_tcp_get_business_card(325): <br>[511] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[512] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[512] MPIR_Init_thread(388)..............: <br>[512] MPID_Init(139).....................: channel initialization failed<br>[512] MPIDI_CH3_Init(38).................: <br>
[512] MPID_nem_init(234).................: <br>[512] MPID_nem_tcp_init(99)..............: <br>[512] MPID_nem_tcp_get_business_card(325): <br>[512] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[515] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[515] MPIR_Init_thread(388)..............: <br>[515] MPID_Init(139).....................: channel initialization failed<br>[515] MPIDI_CH3_Init(38).................: <br>
[515] MPID_nem_init(234).................: <br>[515] MPID_nem_tcp_init(99)..............: <br>[515] MPID_nem_tcp_get_business_card(325): <br>[515] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[443] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>[523] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[523] MPIR_Init_thread(388)..............: <br>
[523] MPID_Init(139).....................: channel initialization failed<br>[523] MPIDI_CH3_Init(38).................: <br>[523] MPID_nem_init(234).................: <br>[523] MPID_nem_tcp_init(99)..............: <br>[523] MPID_nem_tcp_get_business_card(325): <br>
[523] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[443] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[443] MPIR_Init_thread(388)..............: <br>[443] MPID_Init(139).....................: channel initialization failed<br>
[443] MPIDI_CH3_Init(38).................: <br>[443] MPID_nem_init(234).................: <br>[443] MPID_nem_tcp_init(99)..............: <br>[443] MPID_nem_tcp_get_business_card(325): <br>[443] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[518] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[518] MPIR_Init_thread(388)..............: <br>[518] MPID_Init(139).....................: channel initialization failed<br>[518] MPIDI_CH3_Init(38).................: <br>
[518] MPID_nem_init(234).................: <br>[518] MPID_nem_tcp_init(99)..............: <br>[518] MPID_nem_tcp_get_business_card(325): <br>[518] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[444] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[444] MPIR_Init_thread(388)..............: <br>[444] MPID_Init(139).....................: channel initialization failed<br>[444] MPIDI_CH3_Init(38).................: <br>
[444] MPID_nem_init(234).................: <br>[444] MPID_nem_tcp_init(99)..............: <br>[444] MPID_nem_tcp_get_business_card(325): <br>[444] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[624] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>[672] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[446] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>
[626] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>[685] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[440] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>
[628] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>[673] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[446] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[446] MPIR_Init_thread(388)..............: <br>[446] MPID_Init(139).....................: channel initialization failed<br>[446] MPIDI_CH3_Init(38).................: <br>[446] MPID_nem_init(234).................: <br>[446] MPID_nem_tcp_init(99)..............: <br>
[446] MPID_nem_tcp_get_business_card(325): <br>[446] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[630] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>
[682] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[440] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[440] MPIR_Init_thread(388)..............: <br>
[440] MPID_Init(139).....................: channel initialization failed<br>[440] MPIDI_CH3_Init(38).................: <br>[440] MPID_nem_init(234).................: <br>[440] MPID_nem_tcp_init(99)..............: <br>[440] MPID_nem_tcp_get_business_card(325): <br>
[440] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[631] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>[682] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
[682] MPIR_Init_thread(388)..............: <br>[682] MPID_Init(139).....................: channel initialization failed<br>[682] MPIDI_CH3_Init(38).................: <br>[682] MPID_nem_init(234).................: <br>[682] MPID_nem_tcp_init(99)..............: <br>
[682] MPID_nem_tcp_get_business_card(325): <br>[682] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[450] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>
[624] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[624] MPIR_Init_thread(388)..............: <br>[624] MPID_Init(139).....................: channel initialization failed<br>[624] MPIDI_CH3_Init(38).................: <br>
[624] MPID_nem_init(234).................: <br>[624] MPID_nem_tcp_init(99)..............: <br>[624] MPID_nem_tcp_get_business_card(325): <br>[624] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[685] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[685] MPIR_Init_thread(388)..............: <br>[685] MPID_Init(139).....................: channel initialization failed<br>[685] MPIDI_CH3_Init(38).................: <br>
[685] MPID_nem_init(234).................: <br>[685] MPID_nem_tcp_init(99)..............: <br>[685] MPID_nem_tcp_get_business_card(325): <br>[685] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[441] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>[626] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[626] MPIR_Init_thread(388)..............: <br>
[626] MPID_Init(139).....................: channel initialization failed<br>[626] MPIDI_CH3_Init(38).................: <br>[626] MPID_nem_init(234).................: <br>[626] MPID_nem_tcp_init(99)..............: <br>[626] MPID_nem_tcp_get_business_card(325): <br>
[626] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[678] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[447] ifname="<a href="http://stremi-33.reims.grid5000.fr">stremi-33.reims.grid5000.fr</a>"<br>
[628] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[628] MPIR_Init_thread(388)..............: <br>[628] MPID_Init(139).....................: channel initialization failed<br>[628] MPIDI_CH3_Init(38).................: <br>
[628] MPID_nem_init(234).................: <br>[628] MPID_nem_tcp_init(99)..............: <br>[628] MPID_nem_tcp_get_business_card(325): <br>[628] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[687] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[447] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[447] MPIR_Init_thread(388)..............: <br>
[447] MPID_Init(139).....................: channel initialization failed<br>[447] MPIDI_CH3_Init(38).................: <br>[447] MPID_nem_init(234).................: <br>[447] MPID_nem_tcp_init(99)..............: <br>[447] MPID_nem_tcp_get_business_card(325): <br>
[447] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[630] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[630] MPIR_Init_thread(388)..............: <br>[630] MPID_Init(139).....................: channel initialization failed<br>
[630] MPIDI_CH3_Init(38).................: <br>[630] MPID_nem_init(234).................: <br>[630] MPID_nem_tcp_init(99)..............: <br>[630] MPID_nem_tcp_get_business_card(325): <br>[630] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[672] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[672] MPIR_Init_thread(388)..............: <br>[672] MPID_Init(139).....................: channel initialization failed<br>[672] MPIDI_CH3_Init(38).................: <br>
[672] MPID_nem_init(234).................: <br>[672] MPID_nem_tcp_init(99)..............: <br>[672] MPID_nem_tcp_get_business_card(325): <br>[672] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[450] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[450] MPIR_Init_thread(388)..............: <br>[450] MPID_Init(139).....................: channel initialization failed<br>[450] MPIDI_CH3_Init(38).................: <br>
[450] MPID_nem_init(234).................: <br>[450] MPID_nem_tcp_init(99)..............: <br>[450] MPID_nem_tcp_get_business_card(325): <br>[450] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[627] ifname="<a href="http://stremi-27.reims.grid5000.fr">stremi-27.reims.grid5000.fr</a>"<br>[673] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[673] MPIR_Init_thread(388)..............: <br>
[673] MPID_Init(139).....................: channel initialization failed<br>[673] MPIDI_CH3_Init(38).................: <br>[673] MPID_nem_init(234).................: <br>[673] MPID_nem_tcp_init(99)..............: <br>[673] MPID_nem_tcp_get_business_card(325): <br>
[673] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[627] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[627] MPIR_Init_thread(388)..............: <br>[627] MPID_Init(139).....................: channel initialization failed<br>
[627] MPIDI_CH3_Init(38).................: <br>[627] MPID_nem_init(234).................: <br>[627] MPID_nem_tcp_init(99)..............: <br>[627] MPID_nem_tcp_get_business_card(325): <br>[627] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[687] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[687] MPIR_Init_thread(388)..............: <br>[687] MPID_Init(139).....................: channel initialization failed<br>[687] MPIDI_CH3_Init(38).................: <br>
[687] MPID_nem_init(234).................: <br>[687] MPID_nem_tcp_init(99)..............: <br>[687] MPID_nem_tcp_get_business_card(325): <br>[687] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[631] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[631] MPIR_Init_thread(388)..............: <br>[631] MPID_Init(139).....................: channel initialization failed<br>[631] MPIDI_CH3_Init(38).................: <br>
[631] MPID_nem_init(234).................: <br>[631] MPID_nem_tcp_init(99)..............: <br>[631] MPID_nem_tcp_get_business_card(325): <br>[631] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
[683] ifname="<a href="http://stremi-25.reims.grid5000.fr">stremi-25.reims.grid5000.fr</a>"<br>[678] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[678] MPIR_Init_thread(388)..............: <br>
[678] MPID_Init(139).....................: channel initialization failed<br>[678] MPIDI_CH3_Init(38).................: <br>[678] MPID_nem_init(234).................: <br>[678] MPID_nem_tcp_init(99)..............: <br>[678] MPID_nem_tcp_get_business_card(325): <br>
[678] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>[683] Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>[683] MPIR_Init_thread(388)..............: <br>[683] MPID_Init(139).....................: channel initialization failed<br>
[683] MPIDI_CH3_Init(38).................: <br>[683] MPID_nem_init(234).................: <br>[683] MPID_nem_tcp_init(99)..............: <br>[683] MPID_nem_tcp_get_business_card(325): <br>[683] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
<br><br><div class="gmail_quote">On Tue, Aug 2, 2011 at 3:14 PM, Darius Buntinas <span dir="ltr"><<a href="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
OK, can you apply the attached patch, rebuild mpich2 and IMB, then re-run the test with the options that gave the errors?<br>
<br>
The patch should give us more info on the error.<br>
<br>
To apply the patch, do this from the mpich2 source directory:<br>
patch -p0 < dbg.diff<br>
<br>
Then to rebuild mpich2:<br>
make clean<br>
make<br>
make install<br>
<br>
Then, after rebuilding IMB, re-run it like this:<br>
mpiexec -l -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408<br>
<br>
Thanks,<br>
<font color="#888888">-d<br>
<br>
</font><br><br>
<br>
On Aug 2, 2011, at 1:23 PM, teng ma wrote:<br>
<br>
> tma@freims:~$ mpiexec -l -n 2 -binding cpu -f ~/host_mpich env<br>
> [0] SHELL=/bin/bash<br>
> [0] SSH_CLIENT=192.168.159.239 59246 22<br>
> [0] LC_ALL=en_US.UTF-8<br>
> [0] USER=tma<br>
> [0] MAIL=/var/mail/tma<br>
> [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> [0] PWD=/home/tma<br>
> [0] LANG=en_US.UTF-8<br>
> [0] SHLVL=1<br>
> [0] HOME=/home/tma<br>
> [0] LOGNAME=tma<br>
> [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> [0] _=/home/tma/opt/mpi/bin/mpiexec<br>
> [0] TERM=xterm<br>
> [0] OLDPWD=/home/tma/opt/mpi<br>
> [0] SSH_TTY=/dev/pts/26<br>
> [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> [0] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> [0] PMI_RANK=0<br>
> [0] PMI_FD=6<br>
> [0] PMI_SIZE=2<br>
> [1] SHELL=/bin/bash<br>
> [1] SSH_CLIENT=192.168.159.239 59246 22<br>
> [1] LC_ALL=en_US.UTF-8<br>
> [1] USER=tma<br>
> [1] MAIL=/var/mail/tma<br>
> [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> [1] PWD=/home/tma<br>
> [1] LANG=en_US.UTF-8<br>
> [1] SHLVL=1<br>
> [1] HOME=/home/tma<br>
> [1] LOGNAME=tma<br>
> [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> [1] _=/home/tma/opt/mpi/bin/mpiexec<br>
> [1] TERM=xterm<br>
> [1] OLDPWD=/home/tma/opt/mpi<br>
> [1] SSH_TTY=/dev/pts/26<br>
> [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> [1] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> [1] PMI_RANK=1<br>
> [1] PMI_FD=7<br>
> [1] PMI_SIZE=2<br>
><br>
><br>
> and<br>
><br>
><br>
> tma@freims:~$ mpiexec -l -n 2 -f ~/host_mpich env<br>
> [0] SHELL=/bin/bash<br>
> [0] SSH_CLIENT=192.168.159.239 59246 22<br>
> [0] LC_ALL=en_US.UTF-8<br>
> [0] USER=tma<br>
> [0] MAIL=/var/mail/tma<br>
> [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> [0] PWD=/home/tma<br>
> [0] LANG=en_US.UTF-8<br>
> [0] SHLVL=1<br>
> [0] HOME=/home/tma<br>
> [0] LOGNAME=tma<br>
> [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> [0] _=/home/tma/opt/mpi/bin/mpiexec<br>
> [0] TERM=xterm<br>
> [0] OLDPWD=/home/tma/opt/mpi<br>
> [0] SSH_TTY=/dev/pts/26<br>
> [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> [0] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> [0] PMI_RANK=0<br>
> [0] PMI_FD=5<br>
> [0] PMI_SIZE=2<br>
> [1] SHELL=/bin/bash<br>
> [1] SSH_CLIENT=192.168.159.239 59246 22<br>
> [1] LC_ALL=en_US.UTF-8<br>
> [1] USER=tma<br>
> [1] MAIL=/var/mail/tma<br>
> [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin<br>
> [1] PWD=/home/tma<br>
> [1] LANG=en_US.UTF-8<br>
> [1] SHLVL=1<br>
> [1] HOME=/home/tma<br>
> [1] LOGNAME=tma<br>
> [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22<br>
> [1] _=/home/tma/opt/mpi/bin/mpiexec<br>
> [1] TERM=xterm<br>
> [1] OLDPWD=/home/tma/opt/mpi<br>
> [1] SSH_TTY=/dev/pts/26<br>
> [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y<br>
> [1] MPICH_INTERFACE_HOSTNAME=<a href="http://stremi-4.reims.grid5000.fr" target="_blank">stremi-4.reims.grid5000.fr</a><br>
> [1] PMI_RANK=1<br>
> [1] PMI_FD=6<br>
> [1] PMI_SIZE=2<br>
><br>
><br>
><br>
> On Tue, Aug 2, 2011 at 1:49 PM, Darius Buntinas <<a href="mailto:buntinas@mcs.anl.gov">buntinas@mcs.anl.gov</a>> wrote:<br>
><br>
> Can you send us the output of the following?<br>
><br>
> mpiexec -l -n 2 -binding cpu -f ~/host_mpich env<br>
> and<br>
> mpiexec -l -n 2 -f ~/host_mpich env<br>
><br>
> Thanks,<br>
> -d<br>
><br>
> On Aug 2, 2011, at 12:18 PM, teng ma wrote:<br>
><br>
> > If -binding is removed, it's no problem to scale to 768 processes. (32 nodes, 24 core /node). if without binding parameter, what kind of binding strategy mpich2 will use? ( fill out all slots of one nodes, and then another node, or round robin along nodes?)<br>
> ><br>
> > Thanks<br>
> > Teng<br>
> ><br>
> > On Tue, Aug 2, 2011 at 1:14 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>> wrote:<br>
> ><br>
> > Please keep mpich-discuss cc'ed. The below error doesn't seem to be a binding issue. Did you try removing the -binding option to see if it works without that?<br>
> ><br>
> ><br>
> > On 08/02/2011 12:12 PM, teng ma wrote:<br>
> > thanks for the answer. I met another issue with hydra binding. When<br>
> > processes launched exceed 408, it throws error like following:<br>
> ><br>
> ><br>
> > I run it like<br>
> > mpiexec -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> > Fatal error in PMPI_Init_thread: Other MPI error, error stack:<br>
> > MPIR_Init_thread(388)..............:<br>
> > MPID_Init(139).....................: channel initialization failed<br>
> > MPIDI_CH3_Init(38).................:<br>
> > MPID_nem_init(234).................:<br>
> > MPID_nem_tcp_init(99)..............:<br>
> > MPID_nem_tcp_get_business_card(325):<br>
> > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device<br>
> ><br>
> ><br>
> > When processes is less than 407, -binding cpu/rr looks good. If I<br>
> > remove -binding cpu/rr, just with -f ~/host_mpich, it's still ok no<br>
> > matter how many processes. My host_mpich is like:<br>
> ><br>
> > <a href="http://stremi-7.reims.grid5000.fr:24" target="_blank">stremi-7.reims.grid5000.fr:24</a> <<a href="http://stremi-7.reims.grid5000.fr:24" target="_blank">http://stremi-7.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-35.reims.grid5000.fr:24" target="_blank">stremi-35.reims.grid5000.fr:24</a> <<a href="http://stremi-35.reims.grid5000.fr:24" target="_blank">http://stremi-35.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-28.reims.grid5000.fr:24" target="_blank">stremi-28.reims.grid5000.fr:24</a> <<a href="http://stremi-28.reims.grid5000.fr:24" target="_blank">http://stremi-28.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-38.reims.grid5000.fr:24" target="_blank">stremi-38.reims.grid5000.fr:24</a> <<a href="http://stremi-38.reims.grid5000.fr:24" target="_blank">http://stremi-38.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-32.reims.grid5000.fr:24" target="_blank">stremi-32.reims.grid5000.fr:24</a> <<a href="http://stremi-32.reims.grid5000.fr:24" target="_blank">http://stremi-32.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-26.reims.grid5000.fr:24" target="_blank">stremi-26.reims.grid5000.fr:24</a> <<a href="http://stremi-26.reims.grid5000.fr:24" target="_blank">http://stremi-26.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-22.reims.grid5000.fr:24" target="_blank">stremi-22.reims.grid5000.fr:24</a> <<a href="http://stremi-22.reims.grid5000.fr:24" target="_blank">http://stremi-22.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-43.reims.grid5000.fr:24" target="_blank">stremi-43.reims.grid5000.fr:24</a> <<a href="http://stremi-43.reims.grid5000.fr:24" target="_blank">http://stremi-43.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-30.reims.grid5000.fr:24" target="_blank">stremi-30.reims.grid5000.fr:24</a> <<a href="http://stremi-30.reims.grid5000.fr:24" target="_blank">http://stremi-30.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-41.reims.grid5000.fr:24" target="_blank">stremi-41.reims.grid5000.fr:24</a> <<a href="http://stremi-41.reims.grid5000.fr:24" target="_blank">http://stremi-41.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-4.reims.grid5000.fr:24" target="_blank">stremi-4.reims.grid5000.fr:24</a> <<a href="http://stremi-4.reims.grid5000.fr:24" target="_blank">http://stremi-4.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-34.reims.grid5000.fr:24" target="_blank">stremi-34.reims.grid5000.fr:24</a> <<a href="http://stremi-34.reims.grid5000.fr:24" target="_blank">http://stremi-34.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-24.reims.grid5000.fr:24" target="_blank">stremi-24.reims.grid5000.fr:24</a> <<a href="http://stremi-24.reims.grid5000.fr:24" target="_blank">http://stremi-24.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-23.reims.grid5000.fr:24" target="_blank">stremi-23.reims.grid5000.fr:24</a> <<a href="http://stremi-23.reims.grid5000.fr:24" target="_blank">http://stremi-23.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-20.reims.grid5000.fr:24" target="_blank">stremi-20.reims.grid5000.fr:24</a> <<a href="http://stremi-20.reims.grid5000.fr:24" target="_blank">http://stremi-20.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-36.reims.grid5000.fr:24" target="_blank">stremi-36.reims.grid5000.fr:24</a> <<a href="http://stremi-36.reims.grid5000.fr:24" target="_blank">http://stremi-36.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-29.reims.grid5000.fr:24" target="_blank">stremi-29.reims.grid5000.fr:24</a> <<a href="http://stremi-29.reims.grid5000.fr:24" target="_blank">http://stremi-29.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-19.reims.grid5000.fr:24" target="_blank">stremi-19.reims.grid5000.fr:24</a> <<a href="http://stremi-19.reims.grid5000.fr:24" target="_blank">http://stremi-19.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-42.reims.grid5000.fr:24" target="_blank">stremi-42.reims.grid5000.fr:24</a> <<a href="http://stremi-42.reims.grid5000.fr:24" target="_blank">http://stremi-42.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-39.reims.grid5000.fr:24" target="_blank">stremi-39.reims.grid5000.fr:24</a> <<a href="http://stremi-39.reims.grid5000.fr:24" target="_blank">http://stremi-39.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-27.reims.grid5000.fr:24" target="_blank">stremi-27.reims.grid5000.fr:24</a> <<a href="http://stremi-27.reims.grid5000.fr:24" target="_blank">http://stremi-27.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-44.reims.grid5000.fr:24" target="_blank">stremi-44.reims.grid5000.fr:24</a> <<a href="http://stremi-44.reims.grid5000.fr:24" target="_blank">http://stremi-44.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-37.reims.grid5000.fr:24" target="_blank">stremi-37.reims.grid5000.fr:24</a> <<a href="http://stremi-37.reims.grid5000.fr:24" target="_blank">http://stremi-37.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-31.reims.grid5000.fr:24" target="_blank">stremi-31.reims.grid5000.fr:24</a> <<a href="http://stremi-31.reims.grid5000.fr:24" target="_blank">http://stremi-31.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-6.reims.grid5000.fr:24" target="_blank">stremi-6.reims.grid5000.fr:24</a> <<a href="http://stremi-6.reims.grid5000.fr:24" target="_blank">http://stremi-6.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-33.reims.grid5000.fr:24" target="_blank">stremi-33.reims.grid5000.fr:24</a> <<a href="http://stremi-33.reims.grid5000.fr:24" target="_blank">http://stremi-33.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-3.reims.grid5000.fr:24" target="_blank">stremi-3.reims.grid5000.fr:24</a> <<a href="http://stremi-3.reims.grid5000.fr:24" target="_blank">http://stremi-3.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-2.reims.grid5000.fr:24" target="_blank">stremi-2.reims.grid5000.fr:24</a> <<a href="http://stremi-2.reims.grid5000.fr:24" target="_blank">http://stremi-2.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-40.reims.grid5000.fr:24" target="_blank">stremi-40.reims.grid5000.fr:24</a> <<a href="http://stremi-40.reims.grid5000.fr:24" target="_blank">http://stremi-40.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-21.reims.grid5000.fr:24" target="_blank">stremi-21.reims.grid5000.fr:24</a> <<a href="http://stremi-21.reims.grid5000.fr:24" target="_blank">http://stremi-21.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-5.reims.grid5000.fr:24" target="_blank">stremi-5.reims.grid5000.fr:24</a> <<a href="http://stremi-5.reims.grid5000.fr:24" target="_blank">http://stremi-5.reims.grid5000.fr:24</a>><br>
> > <a href="http://stremi-25.reims.grid5000.fr:24" target="_blank">stremi-25.reims.grid5000.fr:24</a> <<a href="http://stremi-25.reims.grid5000.fr:24" target="_blank">http://stremi-25.reims.grid5000.fr:24</a>><br>
> ><br>
> ><br>
> > The configure of mpich2 is just default configure.<br>
> ><br>
> > Thanks<br>
> > Teng<br>
> ><br>
> > On Tue, Aug 2, 2011 at 12:43 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a><br>
> > <mailto:<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>>> wrote:<br>
> ><br>
> ><br>
> > mpiexec -binding rr<br>
> ><br>
> > -- Pavan<br>
> ><br>
> ><br>
> > On 08/02/2011 11:35 AM, teng ma wrote:<br>
> ><br>
> > If I want to do a process-core binding like MVAPICH2's scatter way:<br>
> > assign MPI ranks by nodes in host file, e.g.<br>
> > host1<br>
> > host2<br>
> > host3<br>
> ><br>
> > rank 0 host 1's core 0<br>
> > rank 1 host 2's core 0<br>
> > rank 2 host 3's core 0<br>
> > rank 3 host 1's core 1<br>
> > rank 4 host 2's core 1<br>
> > rank 5 host 3's core 1<br>
> ><br>
> > Is there any easy method in mpich2-1.4 to achieve this binding?<br>
> ><br>
> > Teng Ma<br>
> ><br>
> ><br>
> ><br>
> > _________________________________________________<br>
> > mpich-discuss mailing list<br>
> > <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a> <mailto:<a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a>><br>
> ><br>
> > <a href="https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss</a><br>
> > <<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a>><br>
> ><br>
> ><br>
> > --<br>
> > Pavan Balaji<br>
> > <a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a> <<a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/%7Ebalaji</a>><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > Pavan Balaji<br>
> > <a href="http://www.mcs.anl.gov/%7Ebalaji" target="_blank">http://www.mcs.anl.gov/~balaji</a><br>
> ><br>
> > _______________________________________________<br>
> > mpich-discuss mailing list<br>
> > <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
> > <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
><br>
> _______________________________________________<br>
> mpich-discuss mailing list<br>
> <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
><br>
<br>
<br></blockquote></div><br>