[mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
Jayesh Krishna
jayesh at mcs.anl.gov
Mon Aug 29 11:20:58 CDT 2011
Hi,
Can you make sure that you are using MPICH2's mpiexec to launch IMB (Try specifying the full path to mpiexec)? Also make sure that you have compiled IMB with MPICH2 libs.
Regards,
Jayesh
----- Original Message -----
From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Cc: mpich-discuss at mcs.anl.gov
Sent: Monday, August 29, 2011 8:09:47 AM
Subject: RE: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
Hi,
I get the exact same latencies running mpiexec with or without MPICH_NO_LOCAL set to 1.
I've noticed that the issue only seems to happen with the 64-bit Windows build of MPICH2 1.3.
Running IMB with a 32-bit build of MPICH2 yields 3-4usec latencies...
Am I missing something here?
Cheers,
Youri LACAN-BARTLEY
-----Message d'origine-----
De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
Envoyé : vendredi 26 août 2011 21:00
À : mpich-discuss at mcs.anl.gov
Cc : Youri LACAN-BARTLEY
Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
You can set the MPICH_NO_LOCAL env variable and run the test to use TCP.
mpiexec -n 2 -env MPICH_NO_LOCAL 1 imb.exe PingPong
I get 2-3usec (0-byte IMB) on a Xeon E5520 (2.4 GHz) + 12 GB ram + Windows Server 2008 x64 + no process binding + Nemesis shared memory.
Regards,
Jayesh
----- Original Message -----
From: "Darius Buntinas" <buntinas at mcs.anl.gov>
To: mpich-discuss at mcs.anl.gov
Cc: "Jayesh Krishna" <jayesh at mcs.anl.gov>
Sent: Friday, August 26, 2011 1:13:45 PM
Subject: Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
30us latencies sound like TCP. Can you set the MPICH_NO_LOCAL environment variable to 1 and try it again? I'm not sure how to do that in Windows though (Jayesh?).
If you get the same performance, then for some reason nemesis (or smpd) is not detecting that the processes are on the same node.
-d
On Aug 26, 2011, at 12:28 PM, Youri LACAN-BARTLEY wrote:
> Hi Jayesh,
>
> I've just run a couple different binding schemes and obtain roughly the same latencies whatever core I use within a single socket.
> Running off two different sockets returns the expected result of even higher latencies.
> In this case the processor is a Xeon X7560 (RAM is running at 1066MHz).
>
> On a side note, I've noticed the same behavior on various CPUs.
> What would be a rough average value for intra-node latency using Nemesis' shared memory path?
>
> Youri
>
> -----Message d'origine-----
> De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
> Envoyé : vendredi 26 août 2011 18:19
> À : Youri LACAN-BARTLEY
> Cc : mpich-discuss at mcs.anl.gov
> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>
> Hi,
> Can you try a different binding scheme and see if it makes a difference (cores 1 & 3, cores 1 & 4)? What is the processor that you are running your tests on ?
>
> (PS: Non-shared memory path, Nemesis netmods, would have much higher latencies. So I am assuming that you are using shared memory for MPI communication.)
> Regards,
> Jayesh
>
> ----- Original Message -----
> From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
> To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
> Cc: mpich-discuss at mcs.anl.gov
> Sent: Friday, August 26, 2011 5:19:39 AM
> Subject: RE: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>
> Hi Jayesh,
>
> When running my tests I would systematically make sure MSMPI was using affinity and that MPICH2 had a user defined binding scheme and -channel nemesis explicitly defined.
> Here are the results for a simple IMB Ping Pong test between two cores from the same socket (socket 0, cores 0 and 1):
> (All tests where run on exactly the same hardware)
>
> -- MPICH2 1.3 --
> #---------------------------------------------------
> # Benchmarking PingPong
> # #processes = 2
> # ( 2 additional processes waiting in MPI_Barrier)
> #---------------------------------------------------
> #bytes #repetitions t[usec] Mbytes/sec
> 0 1000 29.77 0.00
> 1 1000 16.31 0.06
> 2 1000 16.00 0.12
> 4 1000 15.34 0.25
> 8 1000 14.99 0.51
> 16 1000 14.83 1.03
> 32 1000 16.64 1.83
> 64 1000 16.39 3.72
> 128 1000 16.47 7.41
> 256 1000 16.76 14.56
> 512 1000 18.99 25.71
> 1024 1000 17.51 55.76
> 2048 1000 20.82 93.80
> 4096 1000 25.89 150.85
> 8192 1000 36.73 212.70
> 16384 1000 54.80 285.13
> 32768 1000 94.50 330.68
> 65536 640 130.02 480.69
> 131072 320 192.06 650.85
> 262144 160 500.97 499.04
> 524288 80 1281.38 390.20
> 1048576 40 2335.72 428.13
> 2097152 20 4581.05 436.58
> 4194304 10 9838.59 406.56
>
> -- MSMPI --
> #---------------------------------------------------
> # Benchmarking PingPong
> # #processes = 2
> #---------------------------------------------------
> #bytes #repetitions t[usec] Mbytes/sec
> 0 1000 1.30 0.00
> 1 1000 1.27 0.75
> 2 1000 1.27 1.50
> 4 1000 1.24 3.08
> 8 1000 1.28 5.96
> 16 1000 1.28 11.95
> 32 1000 1.29 23.60
> 64 1000 1.29 47.18
> 128 1000 1.30 93.96
> 256 1000 1.37 177.74
> 512 1000 1.45 337.71
> 1024 1000 1.71 571.36
> 2048 1000 2.19 891.58
> 4096 1000 4.02 971.25
> 8192 1000 4.60 1696.95
> 16384 1000 6.37 2452.34
> 32768 1000 8.80 3549.73
> 65536 640 14.90 4194.32
> 131072 320 30.11 4151.54
> 262144 160 57.40 4355.55
> 524288 80 110.33 4532.00
> 1048576 40 217.27 4602.61
> 2097152 20 446.33 4480.95
> 4194304 10 977.95 4090.20
>
> As you can see, the average latencies are dramatically different.
> I have no guarantee that MPICH2 is actually using shred memory though.
>
> Cheers,
>
> Youri LACAN-BARTLEY
>
> -----Message d'origine-----
> De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
> Envoyé : jeudi 25 août 2011 17:25
> À : mpich-discuss at mcs.anl.gov
> Cc : Youri LACAN-BARTLEY
> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>
> Hi,
> If you run your MPI job on a single machine/node all communication should go through Nemesis shared memory.
> One important thing to note is that MSMPI binds processes to cores by default and MPICH2 does not. The user is responsible for selecting the binding scheme in MPICH2.
>
> (PS: MPICH2 cannot perform certain default process core bindings like MSMPI because we support older Windows OS versions that MSMPI does not.)
> Regards,
> Jayesh
>
> ----- Original Message -----
> From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
> To: "Rajeev Thakur" <thakur at mcs.anl.gov>
> Cc: mpich-discuss at mcs.anl.gov
> Sent: Thursday, August 25, 2011 8:59:22 AM
> Subject: Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>
> How can I make sure MPICH2 doesn't use TCP but only shared memory? I do specify the use of the nemesis channel on the command line but that hasn't had any impact whatsoever on latency.
> Would you also recommend I build MPICH2 from source? I have no idea how the Windows builds you provide are configured.
>
> Thanks for the help.
>
> Youri LACAN-BARTLEY
>
> -----Message d'origine-----
> De : mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs.anl.gov] De la part de Rajeev Thakur
> Envoyé : jeudi 25 août 2011 15:43
> À : mpich-discuss at mcs.anl.gov
> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>
> One thing to check is whether communication is indeed happening over the Nemesis shared-memory device and not TCP.
>
> Rajeev
>
> On Aug 25, 2011, at 8:40 AM, Youri LACAN-BARTLEY wrote:
>
>> Hi,
>>
>> IMB actually reports massive latencies even with 0 byte messages (roughly 30 usec) which is rather surprising.
>> There is indeed a massive leap in latency going from 8192 byte messages to 16384 byte messages which probably has to do with the missing feature you are referring to.
>> This however doesn't seem to explain with the initial latency is so high..
>>
>> I'll be using MSMPI for now and I'll keep an eye out for any new Windows builds.
>> What feature that's currently in the dev branch should I be looking out for though?
>>
>> Thanks,
>>
>> Youri LACAN-BARTLEY
>>
>> -----Message d'origine-----
>> De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
>> Envoyé : mardi 23 août 2011 19:11
>> À : Youri LACAN-BARTLEY
>> Cc : mpich-discuss at mcs.anl.gov
>> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>> Hi,
>> I am assuming this is due to a missing feature (currently in the dev branch) in the trunk that speeds up large message transfers between MPI processes on a local node. Unfortunately I don't have enough bandwidth to integrate the changes to the trunk right now (It might happen some time in the future).
>> If you are concerned with the performance I would recommend using MSMPI for now.
>>
>> Regards,
>> Jayesh
>>
>> ----- Original Message -----
>> From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
>> To: "Jayesh Krishna" <jayesh at mcs.anl.gov>
>> Cc: mpich-discuss at mcs.anl.gov
>> Sent: Tuesday, August 23, 2011 4:51:15 AM
>> Subject: RE: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>> Hi,
>>
>> Only intra-node communication comes into play here since all four sockets are on the same motherboard.
>> So no IB whatsoever. All communications are via QPI.
>>
>> Cheers,
>>
>> Youri LACAN-BARTLEY
>>
>> -----Message d'origine-----
>> De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
>> Envoyé : lundi 22 août 2011 18:19
>> À : mpich-discuss at mcs.anl.gov
>> Cc : Youri LACAN-BARTLEY
>> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>> Hi,
>> What kind of interconnect do you have ? MPICH2 performs inter-node communication using TCP/IP sockets. There is an experimental support for IB (MSMPI officially supports IB channels), but it is not tested enough (Let me know if you need to try it out).
>>
>> Regards,
>> Jayesh
>>
>> ----- Original Message -----
>> From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
>> To: mpich-discuss at mcs.anl.gov
>> Sent: Friday, August 19, 2011 3:07:56 AM
>> Subject: Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>> I've finally been able to run another series of tests on this server.
>> What I've noticed is that IMB performs very poorly with MPICH2 1.3 be it on 2 or 32 cores (observed latencies are catastrophic).
>> Out of simple curiosity I ran IMB on 32 cores using MSMPI and in this case I obtain perfectly satisfactory performances (I can provide IMB output for both scenarios if necessary).
>>
>> Running our application on eight cores with MPICH2 and binding has yielded the following expected result: 2 cores per socket is 80% faster than running on one single socket.
>>
>> Since MSMPI is based on MPICH2 I'm really surprised by the differences in latency. The only major difference I see is the WinSock Direct protocol used in MSMPI.
>>
>> Would anyone have any kind of idea on what is causing these issues with MPICH2?
>>
>> Thanks,
>>
>> Youri LACAN-BARTLEY
>>
>> -----Message d'origine-----
>> De : mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs..anl.gov] De la part de Youri LACAN-BARTLEY
>> Envoyé : jeudi 12 mai 2011 17:38
>> À : mpich-discuss at mcs.anl.gov
>> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>> Hi Darius,
>>
>> The test machine is currently being used for other purposes but I will run those tests as soon as I can.
>> I wasn't able to run them in my previous tests.
>>
>> I'll post the results as soon as I have them.
>>
>> All the best,
>>
>> Your LACAN-BARTLEY
>>
>> -----Message d'origine-----
>> De : mpich-discuss-bounces at mcs.anl.gov [mailto:mpich-discuss-bounces at mcs..anl.gov] De la part de Darius Buntinas
>> Envoyé : mercredi 27 avril 2011 19:18
>> À : mpich-discuss at mcs.anl.gov
>> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>
>>
>> Can you check that there are no other processes running on your system? Also, see what happens for 31, 30, 29, etc processes to find the point where performance drops suddenly.
>>
>> -d
>>
>>
>> On Apr 27, 2011, at 11:49 AM, Youri LACAN-BARTLEY wrote:
>>
>>> Hi Jayesh,
>>>
>>> First of all, thank you for the swift reply.
>>>
>>> To answer your question, I've been basically binding my mpi jobs with "-binding user:0,1,2,3,[...],29,30,31" since I'm using all available cores on the machine. It's this specific scenario that is bugging me.
>>> I have tried running jobs with let's say 8 cores and specifying different bindings (per core, per socket, shared L2 cache, etc) and the results were what I was expecting.
>>> What I can't explain is why I obtain such a massive speed down between running on 16 cores and 32 cores.
>>> I might be overlooking something but I really can't put my finger on it..
>>>
>>> I've even played around with channels (sock, nemesis, etc) in the hope that this might shed some light on the issue but to no avail.
>>>
>>> If you need more detailed information, don't hesitate to ask.
>>>
>>> Thanks for the help,
>>>
>>> Youri LACAN-BARTLEY
>>>
>>> -----Message d'origine-----
>>> De : Jayesh Krishna [mailto:jayesh at mcs.anl.gov]
>>> Envoyé : mercredi 27 avril 2011 17:30
>>> À : mpich-discuss at mcs.anl.gov
>>> Cc : Youri LACAN-BARTLEY
>>> Objet : Re: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>>
>>> Hi,
>>> What is the binding used ? Did you try different bindings to see if that changes the performance (Does not specifying user defined binding increase/decrease perf)?
>>> More details please.
>>>
>>> -Jayesh
>>>
>>> ----- Original Message -----
>>> From: "Youri LACAN-BARTLEY" <youri.lacan-bartley at transvalor.com>
>>> To: mpich-discuss at mcs.anl.gov
>>> Sent: Wednesday, April 27, 2011 4:04:25 AM
>>> Subject: [mpich-discuss] MPICH2 1.3 speed down on Windows 2008 R2
>>>
>>>
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> I'm currently benchmarking a 32 core machine with four Intel X7560 running Windows 2008 R2.
>>>
>>> I've noticed severe speed down when running on all 32 cores at once using user defined binding and the nemesis channel.
>>>
>>> Would anyone have any idea why this might be the case.
>>>
>>> I've run the exact same hardware with the same software running CentOS 5 and OpenMPI 1.4 and in that case the results show a regular speed up as expected.
>>>
>>>
>>>
>>> Am I hitting a specific MPICH2 issue or has this rather got something to do with Windows?
>>>
>>>
>>>
>>> Kind regards,
>>>
>>>
>>>
>>> Youri LACAN-BARTLEY
>>>
>>>
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>> _______________________________________________
>>> mpich-discuss mailing list
>>> mpich-discuss at mcs.anl.gov
>>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>>
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
>
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
More information about the mpich-discuss
mailing list