[mpich-discuss] a question about process-core binding

teng ma xiaok1981 at gmail.com
Tue Aug 2 14:35:27 CDT 2011 

408 works this time. But when scale to 768, the same error comes out

[29] ifname="stremi-35.reims.grid5000.fr"
[29] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[29] MPIR_Init_thread(388)..............:
[29] MPID_Init(139).....................: channel initialization failed
[29] MPIDI_CH3_Init(38).................:
[29] MPID_nem_init(234).................:
[29] MPID_nem_tcp_init(99)..............:
[29] MPID_nem_tcp_get_business_card(325):
[29] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[27] ifname="stremi-35.reims.grid5000.fr"
[45] ifname="stremi-35.reims.grid5000.fr"
[45] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[45] MPIR_Init_thread(388)..............:
[45] MPID_Init(139).....................: channel initialization failed
[45] MPIDI_CH3_Init(38).................:
[45] MPID_nem_init(234).................:
[45] MPID_nem_tcp_init(99)..............:
[45] MPID_nem_tcp_get_business_card(325):
[45] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[47] ifname="stremi-35.reims.grid5000.fr"
[47] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[47] MPIR_Init_thread(388)..............:
[47] MPID_Init(139).....................: channel initialization failed
[47] MPIDI_CH3_Init(38).................:
[47] MPID_nem_init(234).................:
[47] MPID_nem_tcp_init(99)..............:
[47] MPID_nem_tcp_get_business_card(325):
[47] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[41] ifname="stremi-35.reims.grid5000.fr"
[46] ifname="stremi-35.reims.grid5000.fr"
[41] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[41] MPIR_Init_thread(388)..............:
[41] MPID_Init(139).....................: channel initialization failed
[41] MPIDI_CH3_Init(38).................:
[41] MPID_nem_init(234).................:
[41] MPID_nem_tcp_init(99)..............:
[41] MPID_nem_tcp_get_business_card(325):
[41] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[46] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[46] MPIR_Init_thread(388)..............:
[46] MPID_Init(139).....................: channel initialization failed
[46] MPIDI_CH3_Init(38).................:
[46] MPID_nem_init(234).................:
[46] MPID_nem_tcp_init(99)..............:
[46] MPID_nem_tcp_get_business_card(325):
[46] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[27] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[27] MPIR_Init_thread(388)..............:
[27] MPID_Init(139).....................: channel initialization failed
[27] MPIDI_CH3_Init(38).................:
[27] MPID_nem_init(234).................:
[27] MPID_nem_tcp_init(99)..............:
[27] MPID_nem_tcp_get_business_card(325):
[27] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[120] ifname="stremi-22.reims.grid5000.fr"
[122] ifname="stremi-22.reims.grid5000.fr"
[123] ifname="stremi-22.reims.grid5000.fr"
[127] ifname="stremi-22.reims.grid5000.fr"
[134] ifname="stremi-22.reims.grid5000.fr"
[120] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[120] MPIR_Init_thread(388)..............:
[120] MPID_Init(139).....................: channel initialization failed
[120] MPIDI_CH3_Init(38).................:
[120] MPID_nem_init(234).................:
[120] MPID_nem_tcp_init(99)..............:
[120] MPID_nem_tcp_get_business_card(325):
[120] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[122] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[122] MPIR_Init_thread(388)..............:
[122] MPID_Init(139).....................: channel initialization failed
[122] MPIDI_CH3_Init(38).................:
[122] MPID_nem_init(234).................:
[122] MPID_nem_tcp_init(99)..............:
[122] MPID_nem_tcp_get_business_card(325):
[122] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[123] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[123] MPIR_Init_thread(388)..............:
[123] MPID_Init(139).....................: channel initialization failed
[123] MPIDI_CH3_Init(38).................:
[123] MPID_nem_init(234).................:
[123] MPID_nem_tcp_init(99)..............:
[123] MPID_nem_tcp_get_business_card(325):
[123] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[127] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[127] MPIR_Init_thread(388)..............:
[127] MPID_Init(139).....................: channel initialization failed
[127] MPIDI_CH3_Init(38).................:
[127] MPID_nem_init(234).................:
[127] MPID_nem_tcp_init(99)..............:
[127] MPID_nem_tcp_get_business_card(325):
[127] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[134] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[134] MPIR_Init_thread(388)..............:
[134] MPID_Init(139).....................: channel initialization failed
[134] MPIDI_CH3_Init(38).................:
[134] MPID_nem_init(234).................:
[134] MPID_nem_tcp_init(99)..............:
[134] MPID_nem_tcp_get_business_card(325):
[134] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[125] ifname="stremi-22.reims.grid5000.fr"
[192] ifname="stremi-37.reims.grid5000.fr"
[193] ifname="stremi-37.reims.grid5000.fr"
[195] ifname="stremi-37.reims.grid5000.fr"
[200] ifname="stremi-37.reims.grid5000.fr"
[206] ifname="stremi-37.reims.grid5000.fr"
[211] ifname="stremi-37.reims.grid5000.fr"
[98] ifname="stremi-24.reims.grid5000.fr"
[192] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[192] MPIR_Init_thread(388)..............:
[192] MPID_Init(139).....................: channel initialization failed
[192] MPIDI_CH3_Init(38).................:
[192] MPID_nem_init(234).................:
[192] MPID_nem_tcp_init(99)..............:
[192] MPID_nem_tcp_get_business_card(325):
[192] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[100] ifname="stremi-24.reims.grid5000.fr"
[195] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[195] MPIR_Init_thread(388)..............:
[195] MPID_Init(139).....................: channel initialization failed
[195] MPIDI_CH3_Init(38).................:
[195] MPID_nem_init(234).................:
[195] MPID_nem_tcp_init(99)..............:
[195] MPID_nem_tcp_get_business_card(325):
[195] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[110] ifname="stremi-24.reims.grid5000.fr"
[200] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[200] MPIR_Init_thread(388)..............:
[200] MPID_Init(139).....................: channel initialization failed
[200] MPIDI_CH3_Init(38).................:
[200] MPID_nem_init(234).................:
[200] MPID_nem_tcp_init(99)..............:
[200] MPID_nem_tcp_get_business_card(325):
[200] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[114] ifname="stremi-24.reims.grid5000.fr"
[211] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[211] MPIR_Init_thread(388)..............:
[211] MPID_Init(139).....................: channel initialization failed
[211] MPIDI_CH3_Init(38).................:
[211] MPID_nem_init(234).................:
[211] MPID_nem_tcp_init(99)..............:
[211] MPID_nem_tcp_get_business_card(325):
[211] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[98] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[98] MPIR_Init_thread(388)..............:
[98] MPID_Init(139).....................: channel initialization failed
[98] MPIDI_CH3_Init(38).................:
[98] MPID_nem_init(234).................:
[98] MPID_nem_tcp_init(99)..............:
[98] MPID_nem_tcp_get_business_card(325):
[98] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[193] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[193] MPIR_Init_thread(388)..............:
[193] MPID_Init(139).....................: channel initialization failed
[193] MPIDI_CH3_Init(38).................:
[193] MPID_nem_init(234).................:
[193] MPID_nem_tcp_init(99)..............:
[193] MPID_nem_tcp_get_business_card(325):
[193] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[100] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[100] MPIR_Init_thread(388)..............:
[100] MPID_Init(139).....................: channel initialization failed
[100] MPIDI_CH3_Init(38).................:
[100] MPID_nem_init(234).................:
[100] MPID_nem_tcp_init(99)..............:
[100] MPID_nem_tcp_get_business_card(325):
[100] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[206] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[206] MPIR_Init_thread(388)..............:
[206] MPID_Init(139).....................: channel initialization failed
[206] MPIDI_CH3_Init(38).................:
[206] MPID_nem_init(234).................:
[206] MPID_nem_tcp_init(99)..............:
[206] MPID_nem_tcp_get_business_card(325):
[206] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[110] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[110] MPIR_Init_thread(388)..............:
[110] MPID_Init(139).....................: channel initialization failed
[110] MPIDI_CH3_Init(38).................:
[110] MPID_nem_init(234).................:
[110] MPID_nem_tcp_init(99)..............:
[110] MPID_nem_tcp_get_business_card(325):
[110] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[114] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[114] MPIR_Init_thread(388)..............:
[114] MPID_Init(139).....................: channel initialization failed
[114] MPIDI_CH3_Init(38).................:
[114] MPID_nem_init(234).................:
[114] MPID_nem_tcp_init(99)..............:
[114] MPID_nem_tcp_get_business_card(325):
[114] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[697] ifname="stremi-5.reims.grid5000.fr"
[717] ifname="stremi-5.reims.grid5000.fr"
[504] ifname="stremi-43.reims.grid5000.fr"
[718] ifname="stremi-5.reims.grid5000.fr"
[524] ifname="stremi-43.reims.grid5000.fr"
[711] ifname="stremi-5.reims.grid5000.fr"
[524] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[524] MPIR_Init_thread(388)..............:
[524] MPID_Init(139).....................: channel initialization failed
[524] MPIDI_CH3_Init(38).................:
[524] MPID_nem_init(234).................:
[524] MPID_nem_tcp_init(99)..............:
[524] MPID_nem_tcp_get_business_card(325):
[524] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[697] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[697] MPIR_Init_thread(388)..............:
[697] MPID_Init(139).....................: channel initialization failed
[697] MPIDI_CH3_Init(38).................:
[697] MPID_nem_init(234).................:
[697] MPID_nem_tcp_init(99)..............:
[697] MPID_nem_tcp_get_business_card(325):
[697] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[508] ifname="stremi-43.reims.grid5000.fr"
[711] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[711] MPIR_Init_thread(388)..............:
[711] MPID_Init(139).....................: channel initialization failed
[711] MPIDI_CH3_Init(38).................:
[711] MPID_nem_init(234).................:
[711] MPID_nem_tcp_init(99)..............:
[711] MPID_nem_tcp_get_business_card(325):
[711] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[504] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[504] MPIR_Init_thread(388)..............:
[504] MPID_Init(139).....................: channel initialization failed
[504] MPIDI_CH3_Init(38).................:
[504] MPID_nem_init(234).................:
[504] MPID_nem_tcp_init(99)..............:
[504] MPID_nem_tcp_get_business_card(325):
[504] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[717] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[717] MPIR_Init_thread(388)..............:
[717] MPID_Init(139).....................: channel initialization failed
[717] MPIDI_CH3_Init(38).................:
[717] MPID_nem_init(234).................:
[717] MPID_nem_tcp_init(99)..............:
[717] MPID_nem_tcp_get_business_card(325):
[717] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[508] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[508] MPIR_Init_thread(388)..............:
[508] MPID_Init(139).....................: channel initialization failed
[508] MPIDI_CH3_Init(38).................:
[508] MPID_nem_init(234).................:
[508] MPID_nem_tcp_init(99)..............:
[508] MPID_nem_tcp_get_business_card(325):
[508] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[718] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[718] MPIR_Init_thread(388)..............:
[718] MPID_Init(139).....................: channel initialization failed
[718] MPIDI_CH3_Init(38).................:
[718] MPID_nem_init(234).................:
[718] MPID_nem_tcp_init(99)..............:
[718] MPID_nem_tcp_get_business_card(325):
[718] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[512] ifname="stremi-43.reims.grid5000.fr"
[507] ifname="stremi-43.reims.grid5000.fr"
[444] ifname="stremi-33.reims.grid5000.fr"
[509] ifname="stremi-43.reims.grid5000.fr"
[511] ifname="stremi-43.reims.grid5000.fr"
[515] ifname="stremi-43.reims.grid5000.fr"
[518] ifname="stremi-43.reims.grid5000.fr"
[523] ifname="stremi-43.reims.grid5000.fr"
[507] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[507] MPIR_Init_thread(388)..............:
[507] MPID_Init(139).....................: channel initialization failed
[507] MPIDI_CH3_Init(38).................:
[507] MPID_nem_init(234).................:
[507] MPID_nem_tcp_init(99)..............:
[507] MPID_nem_tcp_get_business_card(325):
[507] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[509] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[509] MPIR_Init_thread(388)..............:
[509] MPID_Init(139).....................: channel initialization failed
[509] MPIDI_CH3_Init(38).................:
[509] MPID_nem_init(234).................:
[509] MPID_nem_tcp_init(99)..............:
[509] MPID_nem_tcp_get_business_card(325):
[509] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[511] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[511] MPIR_Init_thread(388)..............:
[511] MPID_Init(139).....................: channel initialization failed
[511] MPIDI_CH3_Init(38).................:
[511] MPID_nem_init(234).................:
[511] MPID_nem_tcp_init(99)..............:
[511] MPID_nem_tcp_get_business_card(325):
[511] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[512] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[512] MPIR_Init_thread(388)..............:
[512] MPID_Init(139).....................: channel initialization failed
[512] MPIDI_CH3_Init(38).................:
[512] MPID_nem_init(234).................:
[512] MPID_nem_tcp_init(99)..............:
[512] MPID_nem_tcp_get_business_card(325):
[512] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[515] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[515] MPIR_Init_thread(388)..............:
[515] MPID_Init(139).....................: channel initialization failed
[515] MPIDI_CH3_Init(38).................:
[515] MPID_nem_init(234).................:
[515] MPID_nem_tcp_init(99)..............:
[515] MPID_nem_tcp_get_business_card(325):
[515] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[443] ifname="stremi-33.reims.grid5000.fr"
[523] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[523] MPIR_Init_thread(388)..............:
[523] MPID_Init(139).....................: channel initialization failed
[523] MPIDI_CH3_Init(38).................:
[523] MPID_nem_init(234).................:
[523] MPID_nem_tcp_init(99)..............:
[523] MPID_nem_tcp_get_business_card(325):
[523] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[443] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[443] MPIR_Init_thread(388)..............:
[443] MPID_Init(139).....................: channel initialization failed
[443] MPIDI_CH3_Init(38).................:
[443] MPID_nem_init(234).................:
[443] MPID_nem_tcp_init(99)..............:
[443] MPID_nem_tcp_get_business_card(325):
[443] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[518] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[518] MPIR_Init_thread(388)..............:
[518] MPID_Init(139).....................: channel initialization failed
[518] MPIDI_CH3_Init(38).................:
[518] MPID_nem_init(234).................:
[518] MPID_nem_tcp_init(99)..............:
[518] MPID_nem_tcp_get_business_card(325):
[518] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[444] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[444] MPIR_Init_thread(388)..............:
[444] MPID_Init(139).....................: channel initialization failed
[444] MPIDI_CH3_Init(38).................:
[444] MPID_nem_init(234).................:
[444] MPID_nem_tcp_init(99)..............:
[444] MPID_nem_tcp_get_business_card(325):
[444] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[624] ifname="stremi-27.reims.grid5000.fr"
[672] ifname="stremi-25.reims.grid5000.fr"
[446] ifname="stremi-33.reims.grid5000.fr"
[626] ifname="stremi-27.reims.grid5000.fr"
[685] ifname="stremi-25.reims.grid5000.fr"
[440] ifname="stremi-33.reims.grid5000.fr"
[628] ifname="stremi-27.reims.grid5000.fr"
[673] ifname="stremi-25.reims.grid5000.fr"
[446] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[446] MPIR_Init_thread(388)..............:
[446] MPID_Init(139).....................: channel initialization failed
[446] MPIDI_CH3_Init(38).................:
[446] MPID_nem_init(234).................:
[446] MPID_nem_tcp_init(99)..............:
[446] MPID_nem_tcp_get_business_card(325):
[446] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[630] ifname="stremi-27.reims.grid5000.fr"
[682] ifname="stremi-25.reims.grid5000.fr"
[440] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[440] MPIR_Init_thread(388)..............:
[440] MPID_Init(139).....................: channel initialization failed
[440] MPIDI_CH3_Init(38).................:
[440] MPID_nem_init(234).................:
[440] MPID_nem_tcp_init(99)..............:
[440] MPID_nem_tcp_get_business_card(325):
[440] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[631] ifname="stremi-27.reims.grid5000.fr"
[682] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[682] MPIR_Init_thread(388)..............:
[682] MPID_Init(139).....................: channel initialization failed
[682] MPIDI_CH3_Init(38).................:
[682] MPID_nem_init(234).................:
[682] MPID_nem_tcp_init(99)..............:
[682] MPID_nem_tcp_get_business_card(325):
[682] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[450] ifname="stremi-33.reims.grid5000.fr"
[624] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[624] MPIR_Init_thread(388)..............:
[624] MPID_Init(139).....................: channel initialization failed
[624] MPIDI_CH3_Init(38).................:
[624] MPID_nem_init(234).................:
[624] MPID_nem_tcp_init(99)..............:
[624] MPID_nem_tcp_get_business_card(325):
[624] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[685] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[685] MPIR_Init_thread(388)..............:
[685] MPID_Init(139).....................: channel initialization failed
[685] MPIDI_CH3_Init(38).................:
[685] MPID_nem_init(234).................:
[685] MPID_nem_tcp_init(99)..............:
[685] MPID_nem_tcp_get_business_card(325):
[685] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[441] ifname="stremi-33.reims.grid5000.fr"
[626] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[626] MPIR_Init_thread(388)..............:
[626] MPID_Init(139).....................: channel initialization failed
[626] MPIDI_CH3_Init(38).................:
[626] MPID_nem_init(234).................:
[626] MPID_nem_tcp_init(99)..............:
[626] MPID_nem_tcp_get_business_card(325):
[626] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[678] ifname="stremi-25.reims.grid5000.fr"
[447] ifname="stremi-33.reims.grid5000.fr"
[628] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[628] MPIR_Init_thread(388)..............:
[628] MPID_Init(139).....................: channel initialization failed
[628] MPIDI_CH3_Init(38).................:
[628] MPID_nem_init(234).................:
[628] MPID_nem_tcp_init(99)..............:
[628] MPID_nem_tcp_get_business_card(325):
[628] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[687] ifname="stremi-25.reims.grid5000.fr"
[447] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[447] MPIR_Init_thread(388)..............:
[447] MPID_Init(139).....................: channel initialization failed
[447] MPIDI_CH3_Init(38).................:
[447] MPID_nem_init(234).................:
[447] MPID_nem_tcp_init(99)..............:
[447] MPID_nem_tcp_get_business_card(325):
[447] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[630] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[630] MPIR_Init_thread(388)..............:
[630] MPID_Init(139).....................: channel initialization failed
[630] MPIDI_CH3_Init(38).................:
[630] MPID_nem_init(234).................:
[630] MPID_nem_tcp_init(99)..............:
[630] MPID_nem_tcp_get_business_card(325):
[630] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[672] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[672] MPIR_Init_thread(388)..............:
[672] MPID_Init(139).....................: channel initialization failed
[672] MPIDI_CH3_Init(38).................:
[672] MPID_nem_init(234).................:
[672] MPID_nem_tcp_init(99)..............:
[672] MPID_nem_tcp_get_business_card(325):
[672] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[450] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[450] MPIR_Init_thread(388)..............:
[450] MPID_Init(139).....................: channel initialization failed
[450] MPIDI_CH3_Init(38).................:
[450] MPID_nem_init(234).................:
[450] MPID_nem_tcp_init(99)..............:
[450] MPID_nem_tcp_get_business_card(325):
[450] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[627] ifname="stremi-27.reims.grid5000.fr"
[673] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[673] MPIR_Init_thread(388)..............:
[673] MPID_Init(139).....................: channel initialization failed
[673] MPIDI_CH3_Init(38).................:
[673] MPID_nem_init(234).................:
[673] MPID_nem_tcp_init(99)..............:
[673] MPID_nem_tcp_get_business_card(325):
[673] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[627] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[627] MPIR_Init_thread(388)..............:
[627] MPID_Init(139).....................: channel initialization failed
[627] MPIDI_CH3_Init(38).................:
[627] MPID_nem_init(234).................:
[627] MPID_nem_tcp_init(99)..............:
[627] MPID_nem_tcp_get_business_card(325):
[627] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[687] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[687] MPIR_Init_thread(388)..............:
[687] MPID_Init(139).....................: channel initialization failed
[687] MPIDI_CH3_Init(38).................:
[687] MPID_nem_init(234).................:
[687] MPID_nem_tcp_init(99)..............:
[687] MPID_nem_tcp_get_business_card(325):
[687] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[631] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[631] MPIR_Init_thread(388)..............:
[631] MPID_Init(139).....................: channel initialization failed
[631] MPIDI_CH3_Init(38).................:
[631] MPID_nem_init(234).................:
[631] MPID_nem_tcp_init(99)..............:
[631] MPID_nem_tcp_get_business_card(325):
[631] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[683] ifname="stremi-25.reims.grid5000.fr"
[678] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[678] MPIR_Init_thread(388)..............:
[678] MPID_Init(139).....................: channel initialization failed
[678] MPIDI_CH3_Init(38).................:
[678] MPID_nem_init(234).................:
[678] MPID_nem_tcp_init(99)..............:
[678] MPID_nem_tcp_get_business_card(325):
[678] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device
[683] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
[683] MPIR_Init_thread(388)..............:
[683] MPID_Init(139).....................: channel initialization failed
[683] MPIDI_CH3_Init(38).................:
[683] MPID_nem_init(234).................:
[683] MPID_nem_tcp_init(99)..............:
[683] MPID_nem_tcp_get_business_card(325):
[683] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
device


On Tue, Aug 2, 2011 at 3:14 PM, Darius Buntinas <buntinas at mcs.anl.gov>wrote:

> OK, can you apply the attached patch, rebuild mpich2 and IMB, then re-run
> the test with the options that gave the errors?
>
> The patch should give us more info on the error.
>
> To apply the patch, do this from the mpich2 source directory:
>  patch -p0 < dbg.diff
>
> Then to rebuild mpich2:
>  make clean
>  make
>  make install
>
> Then, after rebuilding IMB, re-run it like this:
>  mpiexec -l -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408
>
> Thanks,
> -d
>
>
>
>
> On Aug 2, 2011, at 1:23 PM, teng ma wrote:
>
> > tma at freims:~$ mpiexec -l -n 2 -binding cpu -f ~/host_mpich env
> > [0] SHELL=/bin/bash
> > [0] SSH_CLIENT=192.168.159.239 59246 22
> > [0] LC_ALL=en_US.UTF-8
> > [0] USER=tma
> > [0] MAIL=/var/mail/tma
> > [0]
> PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [0] PWD=/home/tma
> > [0] LANG=en_US.UTF-8
> > [0] SHLVL=1
> > [0] HOME=/home/tma
> > [0] LOGNAME=tma
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [0] _=/home/tma/opt/mpi/bin/mpiexec
> > [0] TERM=xterm
> > [0] OLDPWD=/home/tma/opt/mpi
> > [0] SSH_TTY=/dev/pts/26
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [0] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [0] PMI_RANK=0
> > [0] PMI_FD=6
> > [0] PMI_SIZE=2
> > [1] SHELL=/bin/bash
> > [1] SSH_CLIENT=192.168.159.239 59246 22
> > [1] LC_ALL=en_US.UTF-8
> > [1] USER=tma
> > [1] MAIL=/var/mail/tma
> > [1]
> PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [1] PWD=/home/tma
> > [1] LANG=en_US.UTF-8
> > [1] SHLVL=1
> > [1] HOME=/home/tma
> > [1] LOGNAME=tma
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [1] _=/home/tma/opt/mpi/bin/mpiexec
> > [1] TERM=xterm
> > [1] OLDPWD=/home/tma/opt/mpi
> > [1] SSH_TTY=/dev/pts/26
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [1] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [1] PMI_RANK=1
> > [1] PMI_FD=7
> > [1] PMI_SIZE=2
> >
> >
> > and
> >
> >
> > tma at freims:~$ mpiexec -l -n 2 -f ~/host_mpich env
> > [0] SHELL=/bin/bash
> > [0] SSH_CLIENT=192.168.159.239 59246 22
> > [0] LC_ALL=en_US.UTF-8
> > [0] USER=tma
> > [0] MAIL=/var/mail/tma
> > [0]
> PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [0] PWD=/home/tma
> > [0] LANG=en_US.UTF-8
> > [0] SHLVL=1
> > [0] HOME=/home/tma
> > [0] LOGNAME=tma
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [0] _=/home/tma/opt/mpi/bin/mpiexec
> > [0] TERM=xterm
> > [0] OLDPWD=/home/tma/opt/mpi
> > [0] SSH_TTY=/dev/pts/26
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [0] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [0] PMI_RANK=0
> > [0] PMI_FD=5
> > [0] PMI_SIZE=2
> > [1] SHELL=/bin/bash
> > [1] SSH_CLIENT=192.168.159.239 59246 22
> > [1] LC_ALL=en_US.UTF-8
> > [1] USER=tma
> > [1] MAIL=/var/mail/tma
> > [1]
> PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [1] PWD=/home/tma
> > [1] LANG=en_US.UTF-8
> > [1] SHLVL=1
> > [1] HOME=/home/tma
> > [1] LOGNAME=tma
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [1] _=/home/tma/opt/mpi/bin/mpiexec
> > [1] TERM=xterm
> > [1] OLDPWD=/home/tma/opt/mpi
> > [1] SSH_TTY=/dev/pts/26
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [1] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [1] PMI_RANK=1
> > [1] PMI_FD=6
> > [1] PMI_SIZE=2
> >
> >
> >
> > On Tue, Aug 2, 2011 at 1:49 PM, Darius Buntinas <buntinas at mcs.anl.gov>
> wrote:
> >
> > Can you send us the output of the following?
> >
> >    mpiexec -l -n 2 -binding cpu -f ~/host_mpich env
> > and
> >    mpiexec -l -n 2 -f ~/host_mpich env
> >
> > Thanks,
> > -d
> >
> > On Aug 2, 2011, at 12:18 PM, teng ma wrote:
> >
> > > If -binding is removed, it's no problem to scale to 768 processes. (32
> nodes, 24 core /node). if without binding parameter, what kind of binding
> strategy mpich2 will use? ( fill out all slots of one nodes, and then
> another node,   or round robin along nodes?)
> > >
> > > Thanks
> > > Teng
> > >
> > > On Tue, Aug 2, 2011 at 1:14 PM, Pavan Balaji <balaji at mcs.anl.gov>
> wrote:
> > >
> > > Please keep mpich-discuss cc'ed. The below error doesn't seem to be a
> binding issue. Did you try removing the -binding option to see if it works
> without that?
> > >
> > >
> > > On 08/02/2011 12:12 PM, teng ma wrote:
> > > thanks for the answer. I met another issue with hydra binding. When
> > > processes launched exceed 408,  it throws error like following:
> > >
> > >
> > > I run it like
> > > mpiexec -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such
> device
> > >
> > >
> > > When processes is less than 407, -binding cpu/rr looks good.   If I
> > > remove -binding cpu/rr, just with -f ~/host_mpich, it's still ok no
> > > matter how many processes. My host_mpich is like:
> > >
> > > stremi-7.reims.grid5000.fr:24 <http://stremi-7.reims.grid5000.fr:24>
> > > stremi-35.reims.grid5000.fr:24 <http://stremi-35.reims.grid5000.fr:24>
> > > stremi-28.reims.grid5000.fr:24 <http://stremi-28.reims.grid5000.fr:24>
> > > stremi-38.reims.grid5000.fr:24 <http://stremi-38.reims.grid5000.fr:24>
> > > stremi-32.reims.grid5000.fr:24 <http://stremi-32.reims.grid5000.fr:24>
> > > stremi-26.reims.grid5000.fr:24 <http://stremi-26.reims.grid5000.fr:24>
> > > stremi-22.reims.grid5000.fr:24 <http://stremi-22.reims.grid5000.fr:24>
> > > stremi-43.reims.grid5000.fr:24 <http://stremi-43.reims.grid5000.fr:24>
> > > stremi-30.reims.grid5000.fr:24 <http://stremi-30.reims.grid5000.fr:24>
> > > stremi-41.reims.grid5000.fr:24 <http://stremi-41.reims.grid5000.fr:24>
> > > stremi-4.reims.grid5000.fr:24 <http://stremi-4.reims.grid5000.fr:24>
> > > stremi-34.reims.grid5000.fr:24 <http://stremi-34.reims.grid5000.fr:24>
> > > stremi-24.reims.grid5000.fr:24 <http://stremi-24.reims.grid5000.fr:24>
> > > stremi-23.reims.grid5000.fr:24 <http://stremi-23.reims.grid5000.fr:24>
> > > stremi-20.reims.grid5000.fr:24 <http://stremi-20.reims.grid5000.fr:24>
> > > stremi-36.reims.grid5000.fr:24 <http://stremi-36.reims.grid5000.fr:24>
> > > stremi-29.reims.grid5000.fr:24 <http://stremi-29.reims.grid5000.fr:24>
> > > stremi-19.reims.grid5000.fr:24 <http://stremi-19.reims.grid5000.fr:24>
> > > stremi-42.reims.grid5000.fr:24 <http://stremi-42.reims.grid5000.fr:24>
> > > stremi-39.reims.grid5000.fr:24 <http://stremi-39.reims.grid5000.fr:24>
> > > stremi-27.reims.grid5000.fr:24 <http://stremi-27.reims.grid5000.fr:24>
> > > stremi-44.reims.grid5000.fr:24 <http://stremi-44.reims.grid5000.fr:24>
> > > stremi-37.reims.grid5000.fr:24 <http://stremi-37.reims.grid5000.fr:24>
> > > stremi-31.reims.grid5000.fr:24 <http://stremi-31.reims.grid5000.fr:24>
> > > stremi-6.reims.grid5000.fr:24 <http://stremi-6.reims.grid5000.fr:24>
> > > stremi-33.reims.grid5000.fr:24 <http://stremi-33.reims.grid5000.fr:24>
> > > stremi-3.reims.grid5000.fr:24 <http://stremi-3.reims.grid5000.fr:24>
> > > stremi-2.reims.grid5000.fr:24 <http://stremi-2.reims.grid5000.fr:24>
> > > stremi-40.reims.grid5000.fr:24 <http://stremi-40.reims.grid5000.fr:24>
> > > stremi-21.reims.grid5000.fr:24 <http://stremi-21.reims.grid5000.fr:24>
> > > stremi-5.reims.grid5000.fr:24 <http://stremi-5.reims.grid5000.fr:24>
> > > stremi-25.reims.grid5000.fr:24 <http://stremi-25.reims.grid5000.fr:24>
> > >
> > >
> > > The configure of mpich2 is just default configure.
> > >
> > > Thanks
> > > Teng
> > >
> > > On Tue, Aug 2, 2011 at 12:43 PM, Pavan Balaji <balaji at mcs.anl.gov
> > > <mailto:balaji at mcs.anl.gov>> wrote:
> > >
> > >
> > >    mpiexec -binding rr
> > >
> > >      -- Pavan
> > >
> > >
> > >    On 08/02/2011 11:35 AM, teng ma wrote:
> > >
> > >        If I want to do a process-core binding like MVAPICH2's scatter
> way:
> > >        assign MPI ranks by nodes in host file, e.g.
> > >        host1
> > >        host2
> > >        host3
> > >
> > >        rank 0 host 1's core 0
> > >        rank 1 host 2's core 0
> > >        rank 2 host 3's core 0
> > >        rank 3 host 1's core 1
> > >        rank 4 host 2's core 1
> > >        rank 5 host 3's core 1
> > >
> > >        Is there any easy method in mpich2-1.4 to achieve this binding?
> > >
> > >        Teng Ma
> > >
> > >
> > >
> > >        _________________________________________________
> > >        mpich-discuss mailing list
> > >        mpich-discuss at mcs.anl.gov <mailto:mpich-discuss at mcs.anl.gov>
> > >
> > >        https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> > >        <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
> > >
> > >
> > >    --
> > >    Pavan Balaji
> > >    http://www.mcs.anl.gov/~balaji <http://www.mcs.anl.gov/%7Ebalaji>
> > >
> > >
> > >
> > > --
> > > Pavan Balaji
> > > http://www.mcs.anl.gov/~balaji
> > >
> > > _______________________________________________
> > > mpich-discuss mailing list
> > > mpich-discuss at mcs.anl.gov
> > > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >
> > _______________________________________________
> > mpich-discuss mailing list
> > mpich-discuss at mcs.anl.gov
> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20110802/8096446a/attachment-0001.htm>

More information about the mpich-discuss mailing list