[mpich-discuss] a question about process-core binding
Darius Buntinas
buntinas at mcs.anl.gov
Tue Aug 2 15:25:07 CDT 2011
If the DNS server is being overloaded, the following patch should help. Again, apply the patch, make clean ; make ; make install, rebuild IMB and see if it fails.
Thanks,
-d
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dbg2.diff
Type: application/octet-stream
Size: 699 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/mpich-discuss/attachments/20110802/49e78589/attachment-0001.obj>
-------------- next part --------------
On Aug 2, 2011, at 2:35 PM, teng ma wrote:
> 408 works this time. But when scale to 768, the same error comes out
>
> [29] ifname="stremi-35.reims.grid5000.fr"
> [29] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [29] MPIR_Init_thread(388)..............:
> [29] MPID_Init(139).....................: channel initialization failed
> [29] MPIDI_CH3_Init(38).................:
> [29] MPID_nem_init(234).................:
> [29] MPID_nem_tcp_init(99)..............:
> [29] MPID_nem_tcp_get_business_card(325):
> [29] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [27] ifname="stremi-35.reims.grid5000.fr"
> [45] ifname="stremi-35.reims.grid5000.fr"
> [45] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [45] MPIR_Init_thread(388)..............:
> [45] MPID_Init(139).....................: channel initialization failed
> [45] MPIDI_CH3_Init(38).................:
> [45] MPID_nem_init(234).................:
> [45] MPID_nem_tcp_init(99)..............:
> [45] MPID_nem_tcp_get_business_card(325):
> [45] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [47] ifname="stremi-35.reims.grid5000.fr"
> [47] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [47] MPIR_Init_thread(388)..............:
> [47] MPID_Init(139).....................: channel initialization failed
> [47] MPIDI_CH3_Init(38).................:
> [47] MPID_nem_init(234).................:
> [47] MPID_nem_tcp_init(99)..............:
> [47] MPID_nem_tcp_get_business_card(325):
> [47] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [41] ifname="stremi-35.reims.grid5000.fr"
> [46] ifname="stremi-35.reims.grid5000.fr"
> [41] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [41] MPIR_Init_thread(388)..............:
> [41] MPID_Init(139).....................: channel initialization failed
> [41] MPIDI_CH3_Init(38).................:
> [41] MPID_nem_init(234).................:
> [41] MPID_nem_tcp_init(99)..............:
> [41] MPID_nem_tcp_get_business_card(325):
> [41] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [46] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [46] MPIR_Init_thread(388)..............:
> [46] MPID_Init(139).....................: channel initialization failed
> [46] MPIDI_CH3_Init(38).................:
> [46] MPID_nem_init(234).................:
> [46] MPID_nem_tcp_init(99)..............:
> [46] MPID_nem_tcp_get_business_card(325):
> [46] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [27] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [27] MPIR_Init_thread(388)..............:
> [27] MPID_Init(139).....................: channel initialization failed
> [27] MPIDI_CH3_Init(38).................:
> [27] MPID_nem_init(234).................:
> [27] MPID_nem_tcp_init(99)..............:
> [27] MPID_nem_tcp_get_business_card(325):
> [27] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [120] ifname="stremi-22.reims.grid5000.fr"
> [122] ifname="stremi-22.reims.grid5000.fr"
> [123] ifname="stremi-22.reims.grid5000.fr"
> [127] ifname="stremi-22.reims.grid5000.fr"
> [134] ifname="stremi-22.reims.grid5000.fr"
> [120] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [120] MPIR_Init_thread(388)..............:
> [120] MPID_Init(139).....................: channel initialization failed
> [120] MPIDI_CH3_Init(38).................:
> [120] MPID_nem_init(234).................:
> [120] MPID_nem_tcp_init(99)..............:
> [120] MPID_nem_tcp_get_business_card(325):
> [120] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [122] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [122] MPIR_Init_thread(388)..............:
> [122] MPID_Init(139).....................: channel initialization failed
> [122] MPIDI_CH3_Init(38).................:
> [122] MPID_nem_init(234).................:
> [122] MPID_nem_tcp_init(99)..............:
> [122] MPID_nem_tcp_get_business_card(325):
> [122] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [123] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [123] MPIR_Init_thread(388)..............:
> [123] MPID_Init(139).....................: channel initialization failed
> [123] MPIDI_CH3_Init(38).................:
> [123] MPID_nem_init(234).................:
> [123] MPID_nem_tcp_init(99)..............:
> [123] MPID_nem_tcp_get_business_card(325):
> [123] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [127] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [127] MPIR_Init_thread(388)..............:
> [127] MPID_Init(139).....................: channel initialization failed
> [127] MPIDI_CH3_Init(38).................:
> [127] MPID_nem_init(234).................:
> [127] MPID_nem_tcp_init(99)..............:
> [127] MPID_nem_tcp_get_business_card(325):
> [127] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [134] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [134] MPIR_Init_thread(388)..............:
> [134] MPID_Init(139).....................: channel initialization failed
> [134] MPIDI_CH3_Init(38).................:
> [134] MPID_nem_init(234).................:
> [134] MPID_nem_tcp_init(99)..............:
> [134] MPID_nem_tcp_get_business_card(325):
> [134] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [125] ifname="stremi-22.reims.grid5000.fr"
> [192] ifname="stremi-37.reims.grid5000.fr"
> [193] ifname="stremi-37.reims.grid5000.fr"
> [195] ifname="stremi-37.reims.grid5000.fr"
> [200] ifname="stremi-37.reims.grid5000.fr"
> [206] ifname="stremi-37.reims.grid5000.fr"
> [211] ifname="stremi-37.reims.grid5000.fr"
> [98] ifname="stremi-24.reims.grid5000.fr"
> [192] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [192] MPIR_Init_thread(388)..............:
> [192] MPID_Init(139).....................: channel initialization failed
> [192] MPIDI_CH3_Init(38).................:
> [192] MPID_nem_init(234).................:
> [192] MPID_nem_tcp_init(99)..............:
> [192] MPID_nem_tcp_get_business_card(325):
> [192] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [100] ifname="stremi-24.reims.grid5000.fr"
> [195] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [195] MPIR_Init_thread(388)..............:
> [195] MPID_Init(139).....................: channel initialization failed
> [195] MPIDI_CH3_Init(38).................:
> [195] MPID_nem_init(234).................:
> [195] MPID_nem_tcp_init(99)..............:
> [195] MPID_nem_tcp_get_business_card(325):
> [195] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [110] ifname="stremi-24.reims.grid5000.fr"
> [200] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [200] MPIR_Init_thread(388)..............:
> [200] MPID_Init(139).....................: channel initialization failed
> [200] MPIDI_CH3_Init(38).................:
> [200] MPID_nem_init(234).................:
> [200] MPID_nem_tcp_init(99)..............:
> [200] MPID_nem_tcp_get_business_card(325):
> [200] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [114] ifname="stremi-24.reims.grid5000.fr"
> [211] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [211] MPIR_Init_thread(388)..............:
> [211] MPID_Init(139).....................: channel initialization failed
> [211] MPIDI_CH3_Init(38).................:
> [211] MPID_nem_init(234).................:
> [211] MPID_nem_tcp_init(99)..............:
> [211] MPID_nem_tcp_get_business_card(325):
> [211] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [98] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [98] MPIR_Init_thread(388)..............:
> [98] MPID_Init(139).....................: channel initialization failed
> [98] MPIDI_CH3_Init(38).................:
> [98] MPID_nem_init(234).................:
> [98] MPID_nem_tcp_init(99)..............:
> [98] MPID_nem_tcp_get_business_card(325):
> [98] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [193] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [193] MPIR_Init_thread(388)..............:
> [193] MPID_Init(139).....................: channel initialization failed
> [193] MPIDI_CH3_Init(38).................:
> [193] MPID_nem_init(234).................:
> [193] MPID_nem_tcp_init(99)..............:
> [193] MPID_nem_tcp_get_business_card(325):
> [193] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [100] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [100] MPIR_Init_thread(388)..............:
> [100] MPID_Init(139).....................: channel initialization failed
> [100] MPIDI_CH3_Init(38).................:
> [100] MPID_nem_init(234).................:
> [100] MPID_nem_tcp_init(99)..............:
> [100] MPID_nem_tcp_get_business_card(325):
> [100] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [206] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [206] MPIR_Init_thread(388)..............:
> [206] MPID_Init(139).....................: channel initialization failed
> [206] MPIDI_CH3_Init(38).................:
> [206] MPID_nem_init(234).................:
> [206] MPID_nem_tcp_init(99)..............:
> [206] MPID_nem_tcp_get_business_card(325):
> [206] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [110] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [110] MPIR_Init_thread(388)..............:
> [110] MPID_Init(139).....................: channel initialization failed
> [110] MPIDI_CH3_Init(38).................:
> [110] MPID_nem_init(234).................:
> [110] MPID_nem_tcp_init(99)..............:
> [110] MPID_nem_tcp_get_business_card(325):
> [110] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [114] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [114] MPIR_Init_thread(388)..............:
> [114] MPID_Init(139).....................: channel initialization failed
> [114] MPIDI_CH3_Init(38).................:
> [114] MPID_nem_init(234).................:
> [114] MPID_nem_tcp_init(99)..............:
> [114] MPID_nem_tcp_get_business_card(325):
> [114] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [697] ifname="stremi-5.reims.grid5000.fr"
> [717] ifname="stremi-5.reims.grid5000.fr"
> [504] ifname="stremi-43.reims.grid5000.fr"
> [718] ifname="stremi-5.reims.grid5000.fr"
> [524] ifname="stremi-43.reims.grid5000.fr"
> [711] ifname="stremi-5.reims.grid5000.fr"
> [524] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [524] MPIR_Init_thread(388)..............:
> [524] MPID_Init(139).....................: channel initialization failed
> [524] MPIDI_CH3_Init(38).................:
> [524] MPID_nem_init(234).................:
> [524] MPID_nem_tcp_init(99)..............:
> [524] MPID_nem_tcp_get_business_card(325):
> [524] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [697] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [697] MPIR_Init_thread(388)..............:
> [697] MPID_Init(139).....................: channel initialization failed
> [697] MPIDI_CH3_Init(38).................:
> [697] MPID_nem_init(234).................:
> [697] MPID_nem_tcp_init(99)..............:
> [697] MPID_nem_tcp_get_business_card(325):
> [697] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [508] ifname="stremi-43.reims.grid5000.fr"
> [711] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [711] MPIR_Init_thread(388)..............:
> [711] MPID_Init(139).....................: channel initialization failed
> [711] MPIDI_CH3_Init(38).................:
> [711] MPID_nem_init(234).................:
> [711] MPID_nem_tcp_init(99)..............:
> [711] MPID_nem_tcp_get_business_card(325):
> [711] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [504] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [504] MPIR_Init_thread(388)..............:
> [504] MPID_Init(139).....................: channel initialization failed
> [504] MPIDI_CH3_Init(38).................:
> [504] MPID_nem_init(234).................:
> [504] MPID_nem_tcp_init(99)..............:
> [504] MPID_nem_tcp_get_business_card(325):
> [504] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [717] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [717] MPIR_Init_thread(388)..............:
> [717] MPID_Init(139).....................: channel initialization failed
> [717] MPIDI_CH3_Init(38).................:
> [717] MPID_nem_init(234).................:
> [717] MPID_nem_tcp_init(99)..............:
> [717] MPID_nem_tcp_get_business_card(325):
> [717] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [508] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [508] MPIR_Init_thread(388)..............:
> [508] MPID_Init(139).....................: channel initialization failed
> [508] MPIDI_CH3_Init(38).................:
> [508] MPID_nem_init(234).................:
> [508] MPID_nem_tcp_init(99)..............:
> [508] MPID_nem_tcp_get_business_card(325):
> [508] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [718] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [718] MPIR_Init_thread(388)..............:
> [718] MPID_Init(139).....................: channel initialization failed
> [718] MPIDI_CH3_Init(38).................:
> [718] MPID_nem_init(234).................:
> [718] MPID_nem_tcp_init(99)..............:
> [718] MPID_nem_tcp_get_business_card(325):
> [718] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [512] ifname="stremi-43.reims.grid5000.fr"
> [507] ifname="stremi-43.reims.grid5000.fr"
> [444] ifname="stremi-33.reims.grid5000.fr"
> [509] ifname="stremi-43.reims.grid5000.fr"
> [511] ifname="stremi-43.reims.grid5000.fr"
> [515] ifname="stremi-43.reims.grid5000.fr"
> [518] ifname="stremi-43.reims.grid5000.fr"
> [523] ifname="stremi-43.reims.grid5000.fr"
> [507] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [507] MPIR_Init_thread(388)..............:
> [507] MPID_Init(139).....................: channel initialization failed
> [507] MPIDI_CH3_Init(38).................:
> [507] MPID_nem_init(234).................:
> [507] MPID_nem_tcp_init(99)..............:
> [507] MPID_nem_tcp_get_business_card(325):
> [507] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [509] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [509] MPIR_Init_thread(388)..............:
> [509] MPID_Init(139).....................: channel initialization failed
> [509] MPIDI_CH3_Init(38).................:
> [509] MPID_nem_init(234).................:
> [509] MPID_nem_tcp_init(99)..............:
> [509] MPID_nem_tcp_get_business_card(325):
> [509] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [511] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [511] MPIR_Init_thread(388)..............:
> [511] MPID_Init(139).....................: channel initialization failed
> [511] MPIDI_CH3_Init(38).................:
> [511] MPID_nem_init(234).................:
> [511] MPID_nem_tcp_init(99)..............:
> [511] MPID_nem_tcp_get_business_card(325):
> [511] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [512] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [512] MPIR_Init_thread(388)..............:
> [512] MPID_Init(139).....................: channel initialization failed
> [512] MPIDI_CH3_Init(38).................:
> [512] MPID_nem_init(234).................:
> [512] MPID_nem_tcp_init(99)..............:
> [512] MPID_nem_tcp_get_business_card(325):
> [512] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [515] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [515] MPIR_Init_thread(388)..............:
> [515] MPID_Init(139).....................: channel initialization failed
> [515] MPIDI_CH3_Init(38).................:
> [515] MPID_nem_init(234).................:
> [515] MPID_nem_tcp_init(99)..............:
> [515] MPID_nem_tcp_get_business_card(325):
> [515] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [443] ifname="stremi-33.reims.grid5000.fr"
> [523] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [523] MPIR_Init_thread(388)..............:
> [523] MPID_Init(139).....................: channel initialization failed
> [523] MPIDI_CH3_Init(38).................:
> [523] MPID_nem_init(234).................:
> [523] MPID_nem_tcp_init(99)..............:
> [523] MPID_nem_tcp_get_business_card(325):
> [523] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [443] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [443] MPIR_Init_thread(388)..............:
> [443] MPID_Init(139).....................: channel initialization failed
> [443] MPIDI_CH3_Init(38).................:
> [443] MPID_nem_init(234).................:
> [443] MPID_nem_tcp_init(99)..............:
> [443] MPID_nem_tcp_get_business_card(325):
> [443] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [518] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [518] MPIR_Init_thread(388)..............:
> [518] MPID_Init(139).....................: channel initialization failed
> [518] MPIDI_CH3_Init(38).................:
> [518] MPID_nem_init(234).................:
> [518] MPID_nem_tcp_init(99)..............:
> [518] MPID_nem_tcp_get_business_card(325):
> [518] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [444] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [444] MPIR_Init_thread(388)..............:
> [444] MPID_Init(139).....................: channel initialization failed
> [444] MPIDI_CH3_Init(38).................:
> [444] MPID_nem_init(234).................:
> [444] MPID_nem_tcp_init(99)..............:
> [444] MPID_nem_tcp_get_business_card(325):
> [444] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [624] ifname="stremi-27.reims.grid5000.fr"
> [672] ifname="stremi-25.reims.grid5000.fr"
> [446] ifname="stremi-33.reims.grid5000.fr"
> [626] ifname="stremi-27.reims.grid5000.fr"
> [685] ifname="stremi-25.reims.grid5000.fr"
> [440] ifname="stremi-33.reims.grid5000.fr"
> [628] ifname="stremi-27.reims.grid5000.fr"
> [673] ifname="stremi-25.reims.grid5000.fr"
> [446] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [446] MPIR_Init_thread(388)..............:
> [446] MPID_Init(139).....................: channel initialization failed
> [446] MPIDI_CH3_Init(38).................:
> [446] MPID_nem_init(234).................:
> [446] MPID_nem_tcp_init(99)..............:
> [446] MPID_nem_tcp_get_business_card(325):
> [446] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [630] ifname="stremi-27.reims.grid5000.fr"
> [682] ifname="stremi-25.reims.grid5000.fr"
> [440] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [440] MPIR_Init_thread(388)..............:
> [440] MPID_Init(139).....................: channel initialization failed
> [440] MPIDI_CH3_Init(38).................:
> [440] MPID_nem_init(234).................:
> [440] MPID_nem_tcp_init(99)..............:
> [440] MPID_nem_tcp_get_business_card(325):
> [440] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [631] ifname="stremi-27.reims.grid5000.fr"
> [682] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [682] MPIR_Init_thread(388)..............:
> [682] MPID_Init(139).....................: channel initialization failed
> [682] MPIDI_CH3_Init(38).................:
> [682] MPID_nem_init(234).................:
> [682] MPID_nem_tcp_init(99)..............:
> [682] MPID_nem_tcp_get_business_card(325):
> [682] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [450] ifname="stremi-33.reims.grid5000.fr"
> [624] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [624] MPIR_Init_thread(388)..............:
> [624] MPID_Init(139).....................: channel initialization failed
> [624] MPIDI_CH3_Init(38).................:
> [624] MPID_nem_init(234).................:
> [624] MPID_nem_tcp_init(99)..............:
> [624] MPID_nem_tcp_get_business_card(325):
> [624] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [685] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [685] MPIR_Init_thread(388)..............:
> [685] MPID_Init(139).....................: channel initialization failed
> [685] MPIDI_CH3_Init(38).................:
> [685] MPID_nem_init(234).................:
> [685] MPID_nem_tcp_init(99)..............:
> [685] MPID_nem_tcp_get_business_card(325):
> [685] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [441] ifname="stremi-33.reims.grid5000.fr"
> [626] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [626] MPIR_Init_thread(388)..............:
> [626] MPID_Init(139).....................: channel initialization failed
> [626] MPIDI_CH3_Init(38).................:
> [626] MPID_nem_init(234).................:
> [626] MPID_nem_tcp_init(99)..............:
> [626] MPID_nem_tcp_get_business_card(325):
> [626] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [678] ifname="stremi-25.reims.grid5000.fr"
> [447] ifname="stremi-33.reims.grid5000.fr"
> [628] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [628] MPIR_Init_thread(388)..............:
> [628] MPID_Init(139).....................: channel initialization failed
> [628] MPIDI_CH3_Init(38).................:
> [628] MPID_nem_init(234).................:
> [628] MPID_nem_tcp_init(99)..............:
> [628] MPID_nem_tcp_get_business_card(325):
> [628] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [687] ifname="stremi-25.reims.grid5000.fr"
> [447] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [447] MPIR_Init_thread(388)..............:
> [447] MPID_Init(139).....................: channel initialization failed
> [447] MPIDI_CH3_Init(38).................:
> [447] MPID_nem_init(234).................:
> [447] MPID_nem_tcp_init(99)..............:
> [447] MPID_nem_tcp_get_business_card(325):
> [447] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [630] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [630] MPIR_Init_thread(388)..............:
> [630] MPID_Init(139).....................: channel initialization failed
> [630] MPIDI_CH3_Init(38).................:
> [630] MPID_nem_init(234).................:
> [630] MPID_nem_tcp_init(99)..............:
> [630] MPID_nem_tcp_get_business_card(325):
> [630] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [672] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [672] MPIR_Init_thread(388)..............:
> [672] MPID_Init(139).....................: channel initialization failed
> [672] MPIDI_CH3_Init(38).................:
> [672] MPID_nem_init(234).................:
> [672] MPID_nem_tcp_init(99)..............:
> [672] MPID_nem_tcp_get_business_card(325):
> [672] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [450] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [450] MPIR_Init_thread(388)..............:
> [450] MPID_Init(139).....................: channel initialization failed
> [450] MPIDI_CH3_Init(38).................:
> [450] MPID_nem_init(234).................:
> [450] MPID_nem_tcp_init(99)..............:
> [450] MPID_nem_tcp_get_business_card(325):
> [450] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [627] ifname="stremi-27.reims.grid5000.fr"
> [673] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [673] MPIR_Init_thread(388)..............:
> [673] MPID_Init(139).....................: channel initialization failed
> [673] MPIDI_CH3_Init(38).................:
> [673] MPID_nem_init(234).................:
> [673] MPID_nem_tcp_init(99)..............:
> [673] MPID_nem_tcp_get_business_card(325):
> [673] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [627] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [627] MPIR_Init_thread(388)..............:
> [627] MPID_Init(139).....................: channel initialization failed
> [627] MPIDI_CH3_Init(38).................:
> [627] MPID_nem_init(234).................:
> [627] MPID_nem_tcp_init(99)..............:
> [627] MPID_nem_tcp_get_business_card(325):
> [627] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [687] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [687] MPIR_Init_thread(388)..............:
> [687] MPID_Init(139).....................: channel initialization failed
> [687] MPIDI_CH3_Init(38).................:
> [687] MPID_nem_init(234).................:
> [687] MPID_nem_tcp_init(99)..............:
> [687] MPID_nem_tcp_get_business_card(325):
> [687] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [631] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [631] MPIR_Init_thread(388)..............:
> [631] MPID_Init(139).....................: channel initialization failed
> [631] MPIDI_CH3_Init(38).................:
> [631] MPID_nem_init(234).................:
> [631] MPID_nem_tcp_init(99)..............:
> [631] MPID_nem_tcp_get_business_card(325):
> [631] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [683] ifname="stremi-25.reims.grid5000.fr"
> [678] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [678] MPIR_Init_thread(388)..............:
> [678] MPID_Init(139).....................: channel initialization failed
> [678] MPIDI_CH3_Init(38).................:
> [678] MPID_nem_init(234).................:
> [678] MPID_nem_tcp_init(99)..............:
> [678] MPID_nem_tcp_get_business_card(325):
> [678] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> [683] Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> [683] MPIR_Init_thread(388)..............:
> [683] MPID_Init(139).....................: channel initialization failed
> [683] MPIDI_CH3_Init(38).................:
> [683] MPID_nem_init(234).................:
> [683] MPID_nem_tcp_init(99)..............:
> [683] MPID_nem_tcp_get_business_card(325):
> [683] MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
>
>
> On Tue, Aug 2, 2011 at 3:14 PM, Darius Buntinas <buntinas at mcs.anl.gov> wrote:
> OK, can you apply the attached patch, rebuild mpich2 and IMB, then re-run the test with the options that gave the errors?
>
> The patch should give us more info on the error.
>
> To apply the patch, do this from the mpich2 source directory:
> patch -p0 < dbg.diff
>
> Then to rebuild mpich2:
> make clean
> make
> make install
>
> Then, after rebuilding IMB, re-run it like this:
> mpiexec -l -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408
>
> Thanks,
> -d
>
>
>
>
> On Aug 2, 2011, at 1:23 PM, teng ma wrote:
>
> > tma at freims:~$ mpiexec -l -n 2 -binding cpu -f ~/host_mpich env
> > [0] SHELL=/bin/bash
> > [0] SSH_CLIENT=192.168.159.239 59246 22
> > [0] LC_ALL=en_US.UTF-8
> > [0] USER=tma
> > [0] MAIL=/var/mail/tma
> > [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [0] PWD=/home/tma
> > [0] LANG=en_US.UTF-8
> > [0] SHLVL=1
> > [0] HOME=/home/tma
> > [0] LOGNAME=tma
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [0] _=/home/tma/opt/mpi/bin/mpiexec
> > [0] TERM=xterm
> > [0] OLDPWD=/home/tma/opt/mpi
> > [0] SSH_TTY=/dev/pts/26
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [0] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [0] PMI_RANK=0
> > [0] PMI_FD=6
> > [0] PMI_SIZE=2
> > [1] SHELL=/bin/bash
> > [1] SSH_CLIENT=192.168.159.239 59246 22
> > [1] LC_ALL=en_US.UTF-8
> > [1] USER=tma
> > [1] MAIL=/var/mail/tma
> > [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [1] PWD=/home/tma
> > [1] LANG=en_US.UTF-8
> > [1] SHLVL=1
> > [1] HOME=/home/tma
> > [1] LOGNAME=tma
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [1] _=/home/tma/opt/mpi/bin/mpiexec
> > [1] TERM=xterm
> > [1] OLDPWD=/home/tma/opt/mpi
> > [1] SSH_TTY=/dev/pts/26
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [1] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [1] PMI_RANK=1
> > [1] PMI_FD=7
> > [1] PMI_SIZE=2
> >
> >
> > and
> >
> >
> > tma at freims:~$ mpiexec -l -n 2 -f ~/host_mpich env
> > [0] SHELL=/bin/bash
> > [0] SSH_CLIENT=192.168.159.239 59246 22
> > [0] LC_ALL=en_US.UTF-8
> > [0] USER=tma
> > [0] MAIL=/var/mail/tma
> > [0] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [0] PWD=/home/tma
> > [0] LANG=en_US.UTF-8
> > [0] SHLVL=1
> > [0] HOME=/home/tma
> > [0] LOGNAME=tma
> > [0] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [0] _=/home/tma/opt/mpi/bin/mpiexec
> > [0] TERM=xterm
> > [0] OLDPWD=/home/tma/opt/mpi
> > [0] SSH_TTY=/dev/pts/26
> > [0] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [0] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [0] PMI_RANK=0
> > [0] PMI_FD=5
> > [0] PMI_SIZE=2
> > [1] SHELL=/bin/bash
> > [1] SSH_CLIENT=192.168.159.239 59246 22
> > [1] LC_ALL=en_US.UTF-8
> > [1] USER=tma
> > [1] MAIL=/var/mail/tma
> > [1] PATH=/home/tma/opt/bin:/home/tma/opt/mpi/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/grid5000/code/bin
> > [1] PWD=/home/tma
> > [1] LANG=en_US.UTF-8
> > [1] SHLVL=1
> > [1] HOME=/home/tma
> > [1] LOGNAME=tma
> > [1] SSH_CONNECTION=192.168.159.239 59246 172.16.175.100 22
> > [1] _=/home/tma/opt/mpi/bin/mpiexec
> > [1] TERM=xterm
> > [1] OLDPWD=/home/tma/opt/mpi
> > [1] SSH_TTY=/dev/pts/26
> > [1] GFORTRAN_UNBUFFERED_PRECONNECTED=y
> > [1] MPICH_INTERFACE_HOSTNAME=stremi-4.reims.grid5000.fr
> > [1] PMI_RANK=1
> > [1] PMI_FD=6
> > [1] PMI_SIZE=2
> >
> >
> >
> > On Tue, Aug 2, 2011 at 1:49 PM, Darius Buntinas <buntinas at mcs.anl.gov> wrote:
> >
> > Can you send us the output of the following?
> >
> > mpiexec -l -n 2 -binding cpu -f ~/host_mpich env
> > and
> > mpiexec -l -n 2 -f ~/host_mpich env
> >
> > Thanks,
> > -d
> >
> > On Aug 2, 2011, at 12:18 PM, teng ma wrote:
> >
> > > If -binding is removed, it's no problem to scale to 768 processes. (32 nodes, 24 core /node). if without binding parameter, what kind of binding strategy mpich2 will use? ( fill out all slots of one nodes, and then another node, or round robin along nodes?)
> > >
> > > Thanks
> > > Teng
> > >
> > > On Tue, Aug 2, 2011 at 1:14 PM, Pavan Balaji <balaji at mcs.anl.gov> wrote:
> > >
> > > Please keep mpich-discuss cc'ed. The below error doesn't seem to be a binding issue. Did you try removing the -binding option to see if it works without that?
> > >
> > >
> > > On 08/02/2011 12:12 PM, teng ma wrote:
> > > thanks for the answer. I met another issue with hydra binding. When
> > > processes launched exceed 408, it throws error like following:
> > >
> > >
> > > I run it like
> > > mpiexec -n 408 -binding cpu -f ~/host_mpich ./IMB-MPI1 Bcast -npmin 408
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > > Fatal error in PMPI_Init_thread: Other MPI error, error stack:
> > > MPIR_Init_thread(388)..............:
> > > MPID_Init(139).....................: channel initialization failed
> > > MPIDI_CH3_Init(38).................:
> > > MPID_nem_init(234).................:
> > > MPID_nem_tcp_init(99)..............:
> > > MPID_nem_tcp_get_business_card(325):
> > > MPIDI_Get_IP_for_iface(276)........: ioctl failed errno=19 - No such device
> > >
> > >
> > > When processes is less than 407, -binding cpu/rr looks good. If I
> > > remove -binding cpu/rr, just with -f ~/host_mpich, it's still ok no
> > > matter how many processes. My host_mpich is like:
> > >
> > > stremi-7.reims.grid5000.fr:24 <http://stremi-7.reims.grid5000.fr:24>
> > > stremi-35.reims.grid5000.fr:24 <http://stremi-35.reims.grid5000.fr:24>
> > > stremi-28.reims.grid5000.fr:24 <http://stremi-28.reims.grid5000.fr:24>
> > > stremi-38.reims.grid5000.fr:24 <http://stremi-38.reims.grid5000.fr:24>
> > > stremi-32.reims.grid5000.fr:24 <http://stremi-32.reims.grid5000.fr:24>
> > > stremi-26.reims.grid5000.fr:24 <http://stremi-26.reims.grid5000.fr:24>
> > > stremi-22.reims.grid5000.fr:24 <http://stremi-22.reims.grid5000.fr:24>
> > > stremi-43.reims.grid5000.fr:24 <http://stremi-43.reims.grid5000.fr:24>
> > > stremi-30.reims.grid5000.fr:24 <http://stremi-30.reims.grid5000.fr:24>
> > > stremi-41.reims.grid5000.fr:24 <http://stremi-41.reims.grid5000.fr:24>
> > > stremi-4.reims.grid5000.fr:24 <http://stremi-4.reims.grid5000.fr:24>
> > > stremi-34.reims.grid5000.fr:24 <http://stremi-34.reims.grid5000.fr:24>
> > > stremi-24.reims.grid5000.fr:24 <http://stremi-24.reims.grid5000.fr:24>
> > > stremi-23.reims.grid5000.fr:24 <http://stremi-23.reims.grid5000.fr:24>
> > > stremi-20.reims.grid5000.fr:24 <http://stremi-20.reims.grid5000.fr:24>
> > > stremi-36.reims.grid5000.fr:24 <http://stremi-36.reims.grid5000.fr:24>
> > > stremi-29.reims.grid5000.fr:24 <http://stremi-29.reims.grid5000.fr:24>
> > > stremi-19.reims.grid5000.fr:24 <http://stremi-19.reims.grid5000.fr:24>
> > > stremi-42.reims.grid5000.fr:24 <http://stremi-42.reims.grid5000.fr:24>
> > > stremi-39.reims.grid5000.fr:24 <http://stremi-39.reims.grid5000.fr:24>
> > > stremi-27.reims.grid5000.fr:24 <http://stremi-27.reims.grid5000.fr:24>
> > > stremi-44.reims.grid5000.fr:24 <http://stremi-44.reims.grid5000.fr:24>
> > > stremi-37.reims.grid5000.fr:24 <http://stremi-37.reims.grid5000.fr:24>
> > > stremi-31.reims.grid5000.fr:24 <http://stremi-31.reims.grid5000.fr:24>
> > > stremi-6.reims.grid5000.fr:24 <http://stremi-6.reims.grid5000.fr:24>
> > > stremi-33.reims.grid5000.fr:24 <http://stremi-33.reims.grid5000.fr:24>
> > > stremi-3.reims.grid5000.fr:24 <http://stremi-3.reims.grid5000.fr:24>
> > > stremi-2.reims.grid5000.fr:24 <http://stremi-2.reims.grid5000.fr:24>
> > > stremi-40.reims.grid5000.fr:24 <http://stremi-40.reims.grid5000.fr:24>
> > > stremi-21.reims.grid5000.fr:24 <http://stremi-21.reims.grid5000.fr:24>
> > > stremi-5.reims.grid5000.fr:24 <http://stremi-5.reims.grid5000.fr:24>
> > > stremi-25.reims.grid5000.fr:24 <http://stremi-25.reims.grid5000.fr:24>
> > >
> > >
> > > The configure of mpich2 is just default configure.
> > >
> > > Thanks
> > > Teng
> > >
> > > On Tue, Aug 2, 2011 at 12:43 PM, Pavan Balaji <balaji at mcs.anl.gov
> > > <mailto:balaji at mcs.anl.gov>> wrote:
> > >
> > >
> > > mpiexec -binding rr
> > >
> > > -- Pavan
> > >
> > >
> > > On 08/02/2011 11:35 AM, teng ma wrote:
> > >
> > > If I want to do a process-core binding like MVAPICH2's scatter way:
> > > assign MPI ranks by nodes in host file, e.g.
> > > host1
> > > host2
> > > host3
> > >
> > > rank 0 host 1's core 0
> > > rank 1 host 2's core 0
> > > rank 2 host 3's core 0
> > > rank 3 host 1's core 1
> > > rank 4 host 2's core 1
> > > rank 5 host 3's core 1
> > >
> > > Is there any easy method in mpich2-1.4 to achieve this binding?
> > >
> > > Teng Ma
> > >
> > >
> > >
> > > _________________________________________________
> > > mpich-discuss mailing list
> > > mpich-discuss at mcs.anl.gov <mailto:mpich-discuss at mcs.anl.gov>
> > >
> > > https://lists.mcs.anl.gov/__mailman/listinfo/mpich-discuss
> > > <https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss>
> > >
> > >
> > > --
> > > Pavan Balaji
> > > http://www.mcs.anl.gov/~balaji <http://www.mcs.anl.gov/%7Ebalaji>
> > >
> > >
> > >
> > > --
> > > Pavan Balaji
> > > http://www.mcs.anl.gov/~balaji
> > >
> > > _______________________________________________
> > > mpich-discuss mailing list
> > > mpich-discuss at mcs.anl.gov
> > > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >
> > _______________________________________________
> > mpich-discuss mailing list
> > mpich-discuss at mcs.anl.gov
> > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> >
>
>
>
More information about the mpich-discuss
mailing list